Paper deep dive
A Geometric Notion of Causal Probing
Clément Guerner, Tianyu Liu, Anej Svete, Alexander Warstadt, Ryan Cotterell
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 92%
Last extracted: 3/12/2026, 7:48:05 PM
Summary
The paper introduces a geometric and information-theoretic framework for identifying and evaluating linear concept subspaces in language model representation spaces. By defining intrinsic criteria—erasure, encapsulation, containment, and stability—the authors address spurious correlations that plague existing probing methods. They propose a counterfactual unigram distribution to decouple concept-related and non-concept information, enabling causal interventions for controlled text generation.
Entities (5)
Relation Signals (3)
Language Model → encodes → Linear Subspace
confidence 95% · The linear subspace hypothesis states that, in a language model's representation space, all information about a concept... is encoded in a linear subspace.
LEACE → erases → Concept Information
confidence 90% · We find, for verbal-number, that the LEACE method for linear concept erasure (Belrose et al., 2023) yields a one-dimensional concept subspace which... contains a large share of concept information.
Counterfactual Framework → evaluates → Linear Subspace
confidence 90% · We derive four geometric properties within our counterfactual framework that characterize a precise geometric encoding of a concept.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The linear subspace hypothesis (Bolukbasi et al., 2016) states that, in a language model's representation space, all information about a concept such as verbal number is encoded in a linear subspace. Prior work has relied on auxiliary classification tasks to identify and evaluate candidate subspaces that might give support for this hypothesis. We instead give a set of intrinsic criteria which characterize an ideal linear concept subspace and enable us to identify the subspace using only the language model distribution. Our information-theoretic framework accounts for spuriously correlated features in the representation space (Kumar et al., 2022) by reconciling the statistical notion of concept information and the geometric notion of how concepts are encoded in the representation space. As a byproduct of this analysis, we hypothesize a causal process for how a language model might leverage concepts during generation. Empirically, we find that linear concept erasure is successful in erasing most concept information under our framework for verbal number as well as some complex aspect-level sentiment concepts from a restaurant review dataset. Our causal intervention for controlled generation shows that, for at least one concept across two languages models, the concept subspace can be used to manipulate the concept value of the generated word with precision.
Tags
Links
- Source: https://arxiv.org/abs/2307.15054
- Canonical: https://arxiv.org/abs/2307.15054
Trouble viewing inline? Open PDF directly →
Full Text
69,690 characters extracted from source content.
Expand or collapse full text
A Geometric Notion of Causal Probing Clément Guerner Tianyu LiuAnej Svete Alexander WarstadtRyan Cotterell cguerner,tianyu.liu, anej.svete, awarstadt, ryan.cotterell@inf.ethz.ch Abstract The linear subspace hypothesis (Bolukbasi et al., 2016) states that, in a language model’s representation space, all information about a concept such as verbal number is encoded in a linear subspace. Prior work has relied on auxiliary classification tasks to identify and evaluate candidate subspaces that might give support for this hypothesis. We instead give a set of intrinsic criteria which char- acterize an ideal linear concept subspace and enable us to identify the subspace using only the language model distribution. Our information-theoretic framework accounts for spuriously correlated features in the rep- resentation space (Kumar et al., 2022) by reconciling the statistical notion of concept information and the geometric notion of how concepts are encoded in the representation space. As a byproduct of this analysis, we hypothesize a causal process for how a lan- guage model might leverage concepts dur- ing generation. Empirically, we find that lin- ear concept erasure is successful in erasing most concept information under our frame- work for verbal number as well as some com- plex aspect-level sentiment concepts from a restaurant review dataset. Our causal inter- vention for controlled generation shows that, for at least one concept across two languages models, the concept subspace can be used to manipulate the concept value of the gener- ated word with precision. https://github.com/rycolab/ causalgeom 1 Introduction The reliance of language models (LMs) on con- cepts to make predictions—especially linguistic concepts such asverbal-number 1 —is a well- studied phenomenon (Ravfogel et al., 2021; Lasri et al., 2022; Amini et al., 2023; Arora et al., 2024). 1 Throughout the text, we will use a distinguished typeset- ting to refer to concepts. For instance, the concept of a bird is written asbird. Earlier studies on this topic test whether an LM uses the concept ofverbal-numberby giving it a forced choice between a grammatical and an un- grammatical variant of a sentence (Linzen et al., 2016; Marvin and Linzen, 2018; Goldberg, 2019; Lasri et al., 2022). Consider, for example, the sen- tences: (1)a.The kidswalkthe dog. the kid.PLwalk.3PL.PRESthe dog.SG b. ∗ The kidswalksthe dog. the kid.PLwalk.3SG.PRESthe dog.SG Goldberg (2019) shows that LMs can achieve near perfect accuracy when forced to choose be- tween two such variants. This result suggests that LMs make use ofverbal-numberand other con- cepts to perform next-word prediction, but tells us little about how the representation spaces of these models encode such concepts. Our primary contribution is to construct a novel geometric notion of what it means for a neural LM’s representation space 2 to have information about a concept. Following Bolukbasi et al. (2016) and Ravfogel et al. (2022a), we argue that concepts are naturally operationalized bylinearsubspaces. Linear subspaces lend themselves to tractable algorithms, and they have a simple geometric interpretation which makes it possible to erase a concept from a representation. Existing work (Lasri et al., 2022; Ravfogel et al., 2023) has relied onV-information (Xu et al., 2020) to quantify the amount of information in the representation space of a language model, before and after concept erasure. This measure isextrinsicto the language model, in the sense that it relies on a variational familyVof auxiliary classifiers to measure concept information. In contrast, we propose anintrinsic, information-theoretic (Shannon, 1948) definition of information, by which we mean that information is quantified using distributions induced from 2 For now, we define a representation space simply as the d-dimensional vector space that a language model relies on to encode text. We propose a more formal definition in §2. arXiv:2307.15054v4 [cs.CL] 26 Mar 2025 the language model, i.e., without relying on an additional classifier. We show, via an example inspired by Kumar et al. (2022), that a naïve approach to measuring intrinsic information in a subspace falls victim to spurious correlations. Specifically, while a ground truth,causalconcept subspace may exist in the representation space, correlated non-concept features can also contain information about the concept, complicating the task of estimating concept information in either subspace. Our frame- work breaks the dependence between the concept subspace and its orthogonal complement, allowing us tocorrectlycompute information contained in either subspace while marginalizing out the other. This approach is counterfactual in the sense that it creates representations that would not otherwise occur under the language model. Crucially, it allows us to talk about the mutual information between linear subspaces and concepts. We derive four geometric properties within our counterfactual framework that characterize a pre- cise geometric encoding of a concept. First,era- sureis the condition that the orthogonal comple- ment of the concept subspace should containno information about the concept. Second,encapsula- tionstates that projecting a representation onto our concept subspace should preserveallthe informa- tion about the concept. Third,stabilityquantifies the requirement that projection onto the orthogonal complement of our concept subspace should pre- serve non-concept information. Finally,contain- mentensures that the concept subspace does not contain additional information beyond the concept. Empirically,westudylinguisticcon- ceptsverbal-numberinEnglishand grammatical-genderin French.We find, forverbal-number, that the LEACE method for linear concept erasure (Belrose et al., 2023) yields a one-dimensional concept subspace which, according to our novel counterfactual metrics, contains a large share of concept information while leaving non-concept information relatively untouched.We then leverage our intrinsic measure of information to posit a causal graphical model by which a latent concept may govern LM text generation. This model enables us to derive a causal controlled generation method by manipulating the concept component of a representation. And, indeed, we find evidence that it is possible to use a one-dimensional subspace to control the generation behavior of two language models with respect toverbal-number, but not forgrammatical-gender. We test our causal intervention against the CEBaB (Abraham et al., 2022) benchmark, and find our do-intervention improves the performance of INLP (Ravfogel et al., 2020) and LEACE as causal effect estimators. 2 Concepts and Information In this section, we build towards a definition of mutual information between representations and the concept of interest. 2.1 Language Modeling Basics A language model is a probability distributionp LM overΣ ∗ , the Kleene closure over an alphabetΣ. We parameterizep LM in an autoregressive manner (Du et al., 2023) as follows: p LM (x) =p LM (EOS|x) T Y t=1 p LM (x t |x <t )(1) wherex t ∈Σrefers tot-th word 3 in a string x∈Σ ∗ ,x <t represents the firstt−1words ofx, andEOS/∈Σbeing a distinguished end-of-string symbol. Many language models make use of contextual representations, i.e., they encode a textual context x <t as a real-valued column vectorh(x <t )∈R d . Generally,h(x <t )is deterministically computed from the context stringx <t 4 , such that the repre- sentation space of Eq. (1) is defined as H def = n h(x)|x∈Σ ∗ o ⊆R d . 5 (2) 2.2 Language Models and Concepts We now discuss an exact sense in which a language model can be said to encode a concept. First, we define a concept based on the possible values it can take. We formalize this with aconcept set, a finite, non-empty setCwhose elements are those values. For example, we take the concept set for verbal-numberto include three values:sg(e.g., 3 We refer tox∈Σas words for simplicity, even though in the context of language modeling, these are often called subwords, tokens, or symbols. 4 We relax this assumption later on, such thath(x <t ) can be stochastic givenx <t . One example of a language model with stochastic contextual embeddings is Bowman et al. (2016). 5 Despite consisting of real vectors, the cardinality ofHis countablyinfinite, because it contains exactly one element for every string in the countably infinite setΣ ∗ . Thus, summing overHis discrete and does not require integration. walks),pl(e.g.,walk), andn/a(e.g.,consterna- tion). For various reasons, including syncretism (Baerman, 2007), some verbs in English can have ambiguous concept value depending on context. For instance, in the sentenceYou walked to the store,walkedcan besgorpl. We find similar facts for other concepts in different languages, e.g., forgrammatical-genderin French, the adjective marroncan be bothfemandmsc. To relate language models to concept sets, we introduce a probability distributionι(c|x <t ,x).ι tells us the probability that, in the sequential con- textx <t ∈Σ ∗ , wordx∈Σis annotated with the concept valuec∈ C. For now, we make a simplifying assumption thatιis deterministic, i.e., ι(c|x <t ,x)∈ 0,1for allc∈ C,x∈Σ, and x <t ∈Σ ∗ . 6 This assumption will be relaxed in §4 with a stochastic operationalization of concepts. 2.3 Unigram Information To construct a mutual information between the model’s notion of a concept and its contextual representations, we require a joint distribution between a concept-valued random variable and a representation-valued random variable. In order for this estimate to be intrinsic, we obtain this distribution from the language model itself. We begin by defining thejoint induced un- igramdistribution of the language model over words, concepts, and representations in Eq. (3). In words, this distribution tells how frequently each wordx∈Σco-occurs with a concept valuec∈C and a representationh∈H, on average, in a string x∼p LM of lengthT: p u (x,c,h) def = X x∈Σ ∗ p LM (x)(3) P T t=1 ι(c|x <t ,x t )1 n x=x t ∧h=h(x <t ) o T We can now use Eq. (3) to compute our intrinsic measure of concept information in representations: I(C;H) = X c∈C X h∈H p u (c,h) log p u (c,h) p u (c)p u (h) (4) whereCis aC-valued random variable,His aH-valued random variable, andp u (c,h)is 6 To illustrate this formalism, consider the concept verbal-numberand sentences(1-a)and(1-b). The con- cept set forverbal-numberisC=sg,pl,n/a, andι maps as follows, e.g.,ι(sg|The kids,walk) = 0,ι(pl| The kids,walk) = 1. obtained by marginalizing outxfrom Eq. (3). Eq. (4) tells us how much information on average a representationh∈Hencodes about the identity of a conceptc∈C. Next, we define the following conditional mutual information: I(X;H|C) =(5) X c∈C X x∈Σ X h∈H p u (x,h,c) log p u (x,h|c) p u (x|c)p u (h|c) This quantity measures, given a particular concept valuec∈ C, how much additional information about a wordx∈Σis encoded in the model’s representations. Our information-theoretic framework can be gen- eralized to handle different language-generating processes, e.g., different decoding algorithms for language models, or natural text generated by a process other than the language model under study. 3 A Geometric Encoding of Concepts Thelinear subspace hypothesis(Bolukbasi et al., 2016) makes a prediction about how the concept information we quantify in the previous section is represented geometrically in the LM’s represen- tation space. Specifically, it postulates that there exists alinear subspaceS C ⊆R d that contains all of the information about a concept with valuesC. 7 This hypothesis has been tested on various linguis- tic concepts, includingverbal-number(Ravfogel et al., 2021; Lasri et al., 2022; Amini et al., 2023) andgrammatical-gender(Amini et al., 2023). We follow in this vein, and decompose the rep- resentation spaceHinto a concept linear subspace and an orthogonal, non-concept linear subspace. Then, we provide four information-theoretic met- rics that characterize these subspaces in terms of the information that they contain. 3.1 Concept Partition Given a concept setC, we define a partition of a language model’s representation spaceHinto a concept subspaceS C and its orthogonal comple- ment, thenon-concept subspaceS ⊥ C . We refer toP∈R d×d as the orthogonal projection matrix that projects ontoS ⊥ C , i.e.,P h=proj S ⊥ C (h). In turn,I d −Pprojects onto the concept sub- spaceS C with dimensionality|C|−1, such that 7 Note thatS C isnota linear subspace in the linear- algebraic sense because it is countable and thus not closed under scalar multiplication. e(goes) e(go) e(walks) e(walk) number lemmae(goes) e(go) e(walks) e(walk) number lemma h(The kids) h(The kids) Figure 1: Example of erasure of averbal-number subspace, when predicting the next word givenThe kids. The representation space is two-dimensional with they-axis representing the correct subspace encoding the conceptverbal-number, while thex-axis encodes the lemma. Word representations are denoted withe and contextual representation withh. On the left, we have the original representation space, and on the right, we have the space resulting from erasing information in our concept subspace, i.e., setting they-coordinates of all vectors in the space to 0. walks walk goes go sg000.70 pl00.300 Table 1: Hypothetical joint unigram distributionp u (x,c) ofverbal-numberand word. The lemmawalkis only used asplandgoonly assg. (I d −P)h=proj S C (h). We refer to the partition ofR d intoS C andS ⊥ C as aninformation partition. We use Eq. (4) to define the information about the concept encoded in both. Consider, for exam- ple, information inS ⊥ C aboutCon average over textual contexts: I(C;P H) =(6) X c∈C X h∈H p u (c,P h) log p u (c,P h) p u (c)p u (P h) where the language model’s representations are or- thogonally projected ontoS ⊥ C usingP. Eq. (6) relates thegeometricnotion of a linear subspace with theinformation-theoreticnotion of informa- tion. Thus, ifI(C;P H)is low, we can say that Perases a lot of concept information inHby pro- jecting onto the subspaceS ⊥ C . We denoteH ∥ def = (I d −P)h|h∈H,H ⊥ def =P h|h∈H, and refer toH ∥ ,H ⊥ as random variables correspond- ing to contextual representations projected onto concept and non-concept subspaces, respectively. 3.2 The Perils of Correlation Eq. (6) suggests an attractive property we might ask fromP: It should satisfyI (C;P H) = 0, i.e., completely erase the information about the concept by projecting ontoS ⊥ C . However, as we show next, this naïve characterization is flawed. We illustrate this point with a counterexample inspired by Ku- mar et al. (2022), shown in Fig. 1. Intuitively, such a transformation constitutes successful erasure. 8 To the extent that such a subspace exists in reality, finding thePthat erases this subspace seems like the correct objective. Now, consider the hypothetical joint word– concept unigram distributionp u (x,c)in Table 1. Under this distribution, a projection matrixPthat erases the correcty-axis as shown in Fig. 1 isnot the minimizer of Eq. (6). Knowledge of the lemma alone reveals theverbal-number, becauseH ⊥ (x-axis) andH ∥ (y-axis) are heavily correlated. This means thatI(C;H ⊥ ) = 0.88>0in our toy example in Fig. 1. In order to haveI(C;H ⊥ ) = 0, we would need to letP=0, thereby erasing all lemma information as well. Thus, requiringP to satisfyI(C;H ⊥ ) = 0does not characterize successful erasure because it requires removing all spuriously correlated features. 3.3 A Counterfactual Unigram Distribution The underlying problem with the example given in §3.2 is thatH ∥ andH ⊥ have a common cause that introduces a spurious correlation—the Σ ∗ -valued context random variableX <t . This meansI(H ⊥ ;H ∥ )>0, i.e.,H ⊥ andH ∥ arenot statistically independent. We resolve this issue by building a variant of our information-theoretic ob- jective in Eq. (6) thatassumesthese two variables are statisticallyindependent, i.e.,I(H ⊥ ;H ∥ ) = 0. Under this assumption,H ⊥ would contain no information about the concept, and identification ofH ∥ would be possible via mutual information. While this assumption likely never holds for a concept in practice, this does not matter here—we are crafting a metric under which the correct subspace will be optimal. We denote withh ∥ def = (I d −P)handh ⊥ def =P h the projections onto the concept and non-concept subspace forh∈H. Marginalizing with respect to the induced unigram distribution defined in §2, we 8 One might, but probably shouldn’t, refer to the y-axis as thecausalsubspace, in the sense that manipulating the value in that subspace would result in changing precisely the concept encoded by the representation while leaving other aspects intact. arrive at the following unigram distributions: p u (h ⊥ ) def = X h∈H 1h ⊥ =P hp u (h)(7) p u (h ∥ ) def = X h∈H 1h ∥ = (I d −P)hp u (h)(8) We now construct a variant of our in- ducedunigramp u (x,c,h)thatassumes independencebetweenh ⊥ andh ∥ ,i.e., q u (h) =q u (h ⊥ ,h ∥ ) def =p u (h ⊥ )p u (h ∥ ). This counterfactual unigram distributionq u assigns probability mass to(h ⊥ ,h ∥ )pairs which, under p u (h), would have zero probability. q u (x,c,h ∥ ,h ⊥ ) def = X x <t ∈Σ ∗ ι(c|x,x <t )(9) p LM (x|h ∥ ,h ⊥ )p LM (x <t )p u (h ∥ )p u (h ⊥ ) The choice of the name counterfactual, as well as the implications of this decoupling, will be made precise in §4 when we introduce the causal interpretation of the word–concept model. We define thecounterfactual mutual informa- tionbetween the concept and the projection onto the non-concept subspace as I q (C;H ⊥ ) def =(10) X c∈C X h ⊥ ∈H ⊥ q u (c,h ⊥ ) log q u (c,h ⊥ ) q u (c)q u (h ⊥ ) Importantly, Eq. (10) is minimized by the correct subspace in our example in §3.2.Note that I q (C;H ∥ )can also be obtained by marginalizing outh ⊥ instead. Finally, we defineI q (X;H ⊥ |C) by usingq u instead ofp u in Eq. (5). 3.4 Erasure and Encapsulation We now give formal definitions of erasure and en- capsulation based on Eq. (10). These two notions, combined, determine the extent to which a projec- tion matrixPhas decomposed the representation space into concept and non-concept subspaces. Definition 3.1(Counterfactual Erasure).Let H ⊥ def =P H be anR d -valued random variable. An orthogonal projection matrixP∈R d×d is an ε-eraserofCifI q (C;H ⊥ )< ε. Asε→0, the subspaceS ⊥ C characterized by anε-eraserPfor concept setCwith respect toH encodes very little information about the concept. This means that the language model is no longer able to determine the concept value required by the textual context when generating the next word. We now show that given anε-eraserP, projecting onto its orthogonal complement withI d −P preserves nearly all of the information. Definition 3.2(Counterfactual Encapsulation). LetH ∥ def = (I d −P)Hbe anR d -valued random variable. An orthogonal projection matrix I d −P∈R d×d is anε-encapsulatorofCif I q (C;H)−I q (C;H ∥ )< ε. The quantityI q (C;H)−I q (C;H ∥ )is always non-negative due to the data-processing inequal- ity (Cover and Thomas, 2006, §2.8). Encapsula- tion operationalizes the idea that a subspace gives us all the information needed to correctly iden- tify the concept value required by textual con- text. In App. A, we show that the mutual informa- tion can be additively decomposed:I q (C;H) = I q (C;H ⊥ ) + I q (C;H ∥ ). 3.5 Containment and Stability Erasure and encapsulation do not consider the in- formation content of the representation aside from the concept. With perfect erasure and encapsula- tion, the learned orthogonal projection matrixP could erase much of the non-concept related in- formation fromS ⊥ C . Specifically, ifCis encoded non-linearly (Ravfogel et al., 2022b), then erasure via a linear orthogonal projection could require the removal of additional dimensions that also contain non-concept information. Therefore, in the concept erasure literature, tests of successful erasure are paired with a verification that the representations are not otherwise damaged (Kumar et al., 2022; Ravfogel et al., 2020, 2022a,b; Elazar et al., 2021). We, too, need an information-theoretic notion of preservation of non-concept information inH ⊥ . Preserving information about non-concept aspects ofx <t inH ⊥ requires thatH ∥ only capture information about the concept, i.e. that it should be theminimalsubspace that captures C.Containment formalizes this notion by requiring that, conditioned onC,H ∥ contains little information about the next wordX. Definition 3.3(Counterfactual Containment).Let Pbe an eraser for concept setCwith respect toH. LetH ∥ def = (I d −P)Hbe anR d -valued random variable. Then, we say thatPisε-containedwith respect toHandCifI q (X;H ∥ |C)< ε. Lastly, we define stability to measure how much non-concept information about the next word X H X <t (a) X H H ∥ H ⊥ P X <t C (b) X H H ∥ H ⊥ P X <t C (c) X <t H H ∥ H ⊥ P C X (d) Figure 2: Causal graphical models that demonstrate how a concept may have a causal effect on word generation. Circles represent random variables and diamonds represent deterministic variables.X <t ,C,Xrepresent the random variables for the textual context, the underlying concept, and the next word, respectively.H,H ∥ ,H ⊥ are the representation at stept, its concept-related component, and its component whose concept-related information is erased by orthogonal projection matrixP. Fig. 2a shows the traditional autoregressive causal structure for generation. Fig. 2b is our proposed causal structure for generation with aC-valued latent variableC, with the backdoor path fromCtoHshown in blue. Fig. 2c is the causal structure induced by a do-intervention onC. Finally, Fig. 2d is the causal structure implied by Yang and Klein’s (2021) concept-controlled generation approach. ispreservedin the non-concept subspaceH ⊥ . Ideally, this should be as close as possible to the information present in the entire representation space, ignoring the information about the concept. Definition 3.4(Counterfactual Stability).LetP be an eraser for concept setCwith respect toH. LetH ⊥ def =P H be anR d -valued random variable. Then, we say thatPis anε-stabilizerwith respect toHandCifI q (X;H|C)−I q (X;H ⊥ |C)< ε. The data processing inequality once again ensures thatI q (X;H|C)−I q (X;H ⊥ |C)≥0. Containment and stability together characterize the preservationof information not related to concepts. 4 A Causal Graphical Model We now propose a causal structure by which lan- guage models leverage concepts, in the form of a latent variable, in the generation process. We relate this causal structure to the information partition def- initions given in §3. This enables causal controlled generation via a do-intervention (Pearl, 2009) on the concept random variableC. We finish with a discussion of how our causal controlled generation approach improves upon existing approaches. 4.1 Concept as a Latent Variable We illustrate the traditional autoregressive causal structure, based on the model definition put forth in §2.1, in Fig. 2a. TheΣ ∗ -valued random vari- ableX <t represents the textual context that was previously sampled from the model,His the deter- ministic contextual representation, andXthe word which is sampled usingH. To enable controlled generation with respect to the concept, we introduce aC-valued latent variable Cin the generation process, as shown in Fig. 2b. We make two assumptions aboutC. First, we as- sume that the distribution ofCis influenced by the textual contextX <t , and, moreover, thatCis notfullydetermined by the contextx <t , i.e.,Cis stochastic. This assumption is justified by the fact that the concept value of the next word may not be fully determined by the preceding context, as dis- cussed in §2. Second, we assume that the concept is determinedbeforethe word is sampled. This enables controlled generation, as the concept can directly influence the sampled wordx. In doing so, we break away fromι, which deterministically assigned a concept value to a word based on the preceding context. Our two assumptions onChave an important implication:X <t is no longer the only source of stochasticity inH, as in Fig. 2a. Rather, we as- sume that bothX <t as well asCinfluence the representationH, i.e.,h=h(x <t ,c). Although this construction is not the norm in neural language models, it is a minor departure from reality that greatly enables our model. We note that our causal structure in Fig. 2b differs from the high-level causal abstraction pro- posed by Geiger et al. (2023) for concept erasure via linear projection in the representation space. The authors don’t include a concept-valued ran- dom variable in their causal abstraction. Relat- edly, they argue that iterative null space projection (INLP, Ravfogel et al., 2020) attempts to determine whether a concept is used by a model, not how it is used. In §1, we contend that language mod- els rely on some notion of linguistic concepts like verbal-number, since they consistently predict the correct value. Our intrinsic information-theoretic framework helps us identify both whetherandhow a concept encoding is used by a language model because we measure changes in the model’s pre- dictions when projecting representations onto the concept and non-concept subspaces. 4.2 Causal Controlled Generation We now derive a formal relationship between era- sure, encapsulation, stability, containment, and the assumed causal graph in Fig. 2b. First, inspecting Fig. 2b, we see that if we wish to intervene onCto influenceX, there is a single backdoor path from CtoH. As shown in Fig. 2c,interveningonC directly (denoted bydo(C=c)) removes the edge X <t →C, which lets us easily compute the dis- tribution over the next word after intervention as follows p(x|H ⊥ =h ⊥ ,do(C=c))(11) = X g∈H p(x|H=h ⊥ + (I d −P)g)p(g|c) where, as shown in Fig. 2b, we assume thath ⊥ is deterministic given the contextx <t .gis an R d -valued contextual representation that encodes a textual contextx ′ <t with concept valuec. With high probability,h(x <t )andg(x ′ <t )will be differ- ent. This is the logical conclusion of our decision to treath ⊥ andh ∥ as statistically independent—we can intervene on the generation process by setting the value of the concept component independently. We now make good on our decision to name the counterfactual unigram distribution from Eq. (9) as such.Assuming the model Fig. 2b, a do- intervention onC—as depicted in Fig. 2c—implies erasure, encapsulation, stability, and containment. We make this idea formal in the following theorem. Theorem 4.1.Consider a joint distributionpthat factors as in Fig. 2b, parameterized by orthogonal projection matrixP. Under the distribution p do (x,h ⊥ ,h ∥ ,c) =p(x|h ⊥ ,h ∥ )(12) p(h ⊥ |do (C=c))p(h ∥ |do (C=c))p(c) we have thatPis anε-eraser,I d −Pis an ε-encapsulator,I d −Pis anε-container andP is anε-stabilizer for everyε >0. Proof.See App. B.■ What Theorem 4.1 tells us is that the graph given in Fig. 2b is consistent with the technical elabora- tion in §3. Specifically, it means that erasure, en- capsulation, stability, and containment are all prop- erties that we expect a causal distribution resulting from an intervention on a concept to have. The in- terventional distributions, hence, motivate our dis- cussion on independentp(h ∥ )andp(h ⊥ )in §3.3. 4.3 Non-causal Controlled Generation Controlled generation involving the manipulation of concepts is not a new problem. We contextualize our approach relative to Yang and Klein’s (2021) method. They perform controlled generation as follows. First, they train a classifier to predict a concept valuec∈Cfrom the contextual represen- tationhof a language model. Then, they perform controlled generation by conditioning on a concept valueC=cand applying Bayes’ rule as follows: p(x|x <t ,C=c)(13) ∝p(C=c|(I d −P)h(x <t ))p(x|x <t ) We illustrate the causal structure implied by this ap- proach in Fig. 2d. We usePto relate this approach to our subspace formulation, 9 but Yang and Klein (2021) do not make use of concept subspaces. A do-intervention onChas no effect onXwith this causal structure, because there is no causal path fromCtoXin Fig. 2d. This is why the authorsconditiononCinstead. In this sense, Yang and Klein’s (2021) and similar methods are not causal and cannot easily be extended to be so. As discussed in §4.2, our approachiscausal, but such an analysis may come at the price of a number of restricting assumptions that are not fully met in practice. In the next section, we explain how we go about testing these assumptions with data. 5 Experimental Setup In the remainder of the paper, we test our framework empirically. Specifically, we answer three questions.First, are we able to find a projection matrixPthat meets our definitions in §3, across multiple concepts and models? Second, can we use the resulting concept subspace to successfully control the model’s generation behavior, as theorized in §4? Finally, how does our 9 Thus, we assume that the classifier is restricted to looking atH ∥ to make its prediction. do-intervention compare to other concept-based explanation methods on the CEBaB benchmark? 5.1 Linguistic Concepts ConceptsandModels.Weperform ouranalysisontwolinguisticconcepts, verbal-numberin English withC=sg,pl,n/a andgrammatical-genderinFrenchwith C=fem,msc,n/a. For each of these concepts, we study the representation spaces of autoregres- sive language models, namely GPT2 (Radford et al., 2019) and Llama 2 (Touvron et al., 2023). 10 Data.Forverbal-numberin English, we use Linzen et al.’s (2016) number agreement dataset. This dataset consists of sentences from Wikipedia that contain asgorplverb with thefact (ground truth verb) and thefoil(inflected form of the fact to have opposite concept value). For grammatical-genderin French, we rely on three treebanks from Universal Dependencies (Nivre et al., 2020): French GSD (Guillaume et al., 2019), ParTUT (Sanguinetti and Bosco, 2015, 2014; Bosco and Sanguinetti, 2014), and Rhap- sodie (Lacheret et al., 2014). We replicate the pre- processing steps of Linzen et al. (2016) on each of these datasets, i.e., we filter sentences to those containingfemormscnouns with an associated adjective, and we obtain the foil by inflecting the grammatical-genderof this adjective. 5.2 CEBaB Benchmark The CEBaB (Abraham et al., 2022) benchmark dataset consists of original restaurant reviews in English, annotated for their overall sentiment on a 5-star scale, as well as aspect-level sentiment la- bels on four concepts,ambiance,food,noise, and service, with concept valuesC=pos,neg,n/a. The dataset includes human-written counterfactual reviews, where the text of the original review is al- tered to create pairs of reviews that differ according to a single aspect-level concept. We use this dataset in two experiments requiring different setups. In this section, we outline how we adapt the dataset to be able to learnPand compute our information- theoretic framework on the four CEBaB concepts. Later, in §6.3, we also test our do-intervention on the CEBaB benchmark. For that experiment, we 10 We rely on the implementations in the transform- ers library (Wolf et al., 2020), namely:gpt2-large andmeta-llama/Llama-2-7b-hfforverbal-numberand gpt2-base-frenchforgrammatical-gender. refer the reader to Abraham et al. (2022) for ex- periment details, since we simply plugged our do- intervention into their pipeline. Data and Models.For CEBaB aspect-level sen- timent concepts, we use the pre-processing steps in the CEBaB repository 11 , yielding train-exclusive, dev and test data splits. To adapt CEBaB to an autoregressive language modeling setting, we ap- pend a concept-specific prompt at the end of the restaurant review, e.g.,... The food was. This prompt induces the model into predicting either a posornegadjective depending on the contents of the review with respect to that concept. 12 We ran- domly sample one of five concept-related prompts for each review and append it to the end of the review text (see Table 4 in App. C for prompts). Then, we take the last hidden state of the prompt to learnPand compute our counterfactual mutual information. This prompt-based approach enables the use of our framework for a wide range of com- plex concepts. Lastly, we study these four concepts using GPT2 (Radford et al., 2019) and Llama 2 (Touvron et al., 2023), using the same transform- ers implementations as forverbal-number(Wolf et al., 2020). 5.3 Shared Experimental Setup Concept Definition.In §2.2, we defined our context-dependent distributionιas a means of re- lating language models and concepts. In practice, we drop the context-dependent aspect and define a concept via a list of words for each concept value. For linguistic concepts, we construct our lists of words by using SpaCy (Montani et al., 2022) to tag the French and English Wikipedia corpora (Foun- dation, 2023), respectively. Forverbal-number, we use the tagged English words to obtain lists of third person presentsgandplverbs, which we then align to obtain matching pairs, e.g., (walks,walk). The process is the same forgrammatical-gender in French, leading to gendered pairs of adjectives, e.g., (français,française). For CEBaB concepts, we prompt ChatGPT (OpenAI, 2022) to create lists ofposandnegadjectives relating to each concept. In this case, the lists do not lend themselves to pair- ing, but include equal numbers of words for each 11 https://github.com/CEBaBing/CEBaB 12 Initially, we learnedPwithout the prompt, using the last hidden state of the review to replicate Abraham et al.’s (2022) training process forPusing INLP (Ravfogel et al., 2020). Poor erasure performance led us to use prompts for both learning and evaluation. Concept InformationNon-Concept Information ConceptModelI(C;H)Erasure (↑)Corr. Erasure (↑) Encaps. (↑)Reconst. (↑) I(X;H|C)Containment (↑) Stability (↑) number gpt2-large0.50±0.040.78±0.040.69±0.040.52±0.030.74±0.041.02±0.150.87±0.021.00±0.02 number llama20.49±0.060.76±0.050.75±0.040.55±0.050.78±0.081.07±0.070.78±0.011.05±0.02 gender gpt2-base fr 0.46±0.040.57±0.040.49±0.080.34±0.030.77±0.032.02±0.130.93±0.010.95±0.01 ambiance gpt2-large0.25±0.020.84±0.040.97±0.010.09±0.020.25±0.040.41±0.100.77±0.070.76±0.13 ambiance llama20.35±0.030.39±0.040.18±0.050.46±0.031.07±0.060.58±0.130.72±0.060.84±0.07 food gpt2-large0.28±0.020.74±0.040.27±0.120.59±0.040.85±0.060.44±0.090.81±0.030.60±0.09 food llama20.41±0.030.76±0.030.68±0.030.97±0.071.21±0.060.66±0.070.73±0.030.81±0.05 noise gpt2-large0.21±0.010.78±0.010.60±0.030.35±0.050.57±0.050.31±0.110.71±0.140.82±0.10 noise llama20.28±0.030.29±0.080.21±0.100.43±0.081.15±0.050.60±0.100.73±0.050.73±0.09 service gpt2-large0.35±0.030.65±0.030.40±0.110.44±0.050.79±0.050.39±0.110.74±0.080.74±0.14 service llama20.39±0.020.51±0.040.25±0.050.86±0.071.35±0.090.53±0.130.68±0.080.86±0.05 Table 2: Information-theoretic evaluation results. The first set of columns quantifies concept information in concept and non-concept subspaces, with total concept informationI(C;H)alongside theCorrelational Erasure Ratio, and theCounterfactual Erasure,EncapsulationandReconstructedratios. The second set of columns describes the preservation of non-concept information, with total non-concept informationI(X;H|C)alongside theContainment RatioandStability Ratio. We expect values of the ratios to fall within[0,1], where higher is better. Each entry shows standard deviation over random restarts. The first set of rows shows linguistic concepts verbal-numberandgrammatical-gender, split by model. The second set of rows shows CEBaB aspect-level sentiment conceptsambiance,food,noise, andservice, withPtrained on the CEBaB dataset. concept (see Table 5 for word lists). Computing word probabilities.A word can be tokenized to one or more tokens. We assume that the tokenization for each word is unique. When computing the probability of a multi-token word while intervening on the representation space, e.g., q u (x|h ⊥ ), we apply concept erasure only when computing the probability of the first token in the word. Subsequent token probabilities are obtained by recomputinghwithout intervention. We then sum log probabilities of each successive token, as well as the log probability that the token after the word is either the beginning of a new word, punctuation, or theEOStoken. Finding the Concept Subspace.We findP using LEACE (Belrose et al., 2023), the state-of- the-art method for linear concept erasure. LEACE maximizes a cross-entropy loss on samples from ep u with respect toP, which constitutes a lower bound on thecorrelationalI(C;H ⊥ ). Results are reported for threePestimates obtained from randomized train, test splits for each concept, and three random restarts of the experiment for eachP. Estimatingp u (h),p u (h ⊥ ), andp u (h ∥ ).For each model, we generate strings by repeatedly sampling from the language model using ancestral and nucleus sampling, starting withBOS, until the model’s context size limit is reached orEOS is sampled. At each sampling step, we save the pair(x t ,h(x <t ),c t ), wherec t is the concept label of the sample. We then compute sums overHby randomly sampling from our dataset of generated h(x <t )assuming a uniform distribution. We apply Pto geth ⊥ andh ∥ . The concept valuen/a.We excluden/afrom our concept set when learningPand when measuring concept information. We are most interested in the behavior of the model when generating non-n/a words. In the overwhelming majority of textual contexts,n/ais far more likely and erasure is trivial. As such, our train sets for LEACE and test sets for computing our metrics contain only non-n/a context strings. This meansPonly learns to erase the distinction between non-n/aconcept values, so we again excluden/awhen computing concept information metrics of erasure, encapsulation, and partition reconstruction defined in §3.4. We can, however, includen/awords for non-concept information metrics of containment and stability defined in §3.5. We estimate this by randomly sampling and computing the probabilities of3k non-concept words in the concept language with each random restart of our evaluation pipeline. 6 Results 6.1 Partitioning of Concept Information In Table 2, we test empirically whether LEACE (Belrose et al., 2023) yields aPthat performs well according to our counterfactual information- theoretic framework defined in §3. To facilitate the interpretation of our results, we reformulate our definitions into ratios with values between 0 and 1, such that a higher value is better: RatioDefinition Erasure1− I q (C;H ⊥ ) I(C;H) Corr. Erasure1− I(C;H ⊥ ) I(C;H) Encapsulation I q (C;H ∥ ) I(C;H) Reconstructed I q (C;H ∥ )+I q (C;H ⊥ ) I(C;H) Containment1− I q (X;H ∥ |C) I(X;H|C) Stability I q (X;H ⊥ |C) I(X;H|C) Table 2 shows that LEACE can yield an adequate representation space partitioning according to our framework, but does not drive the counterfactual erasure and encapsulation ratios to 1. The mean erasure ratio across all datasets and models is0.64 and the encapsulation ratio is0.50. This indicates that (a) the non-concept representation components H ⊥ still contain some information about the con- cept and (b) the concept representation components H ∥ cannot fully determine the conceptc. Empiri- cally, it is difficult to determine whether these find- ings are due to the LEACE objective falling victim to spurious correlations, as described in §3.2, or to the concept encoding being non-linear. For CEBaB concepts, the correlational erasure ratio is typically far lower than the counterfactual one. This sug- gests that while spuriously correlated information remains inH ⊥ , the actual concept direction has been identified more accurately than a simple cor- relational analysis would imply. A containment ratio less than1indicates thatH ∥ generally contains some non-concept information, with a mean value of0.77. Since all concepts are binary after droppingn/a, the concept subspaces are one-dimensional. The amount of non-concept information contained in a one-dimensional subspace suggests that the concept features are highly correlated with non-concept features. The stability ratio has a mean of0.83, confirming that LEACE does some damage to non-concept information inH ⊥ . We note, however, that for our simple linguistic concepts, this damage is minimal. Lastly, we attribute the failure to learn a concept partition forgrammatical-genderto limitations of the model itself. Compared to English, the best available Frenchgpt2model is trained on less data and has fewer parameters. In our preliminary exper- Orig. Acc. Erased Acc. Do Acc. 0.0 0.2 0.4 0.6 0.8 1.0 Accuracy GPT2-large, Number Orig. Acc. Erased Acc. Do Acc. Llama2, Number Orig. Acc. Erased Acc. Do Acc. GPT2-base-french, Gender Majority Figure 3: Controlled generation experiment. Reported values are computed on (context, fact, foil) samples from the test split of our curated datasets of natural text used to train LEACE.Orig. Accrefers to the accu- racy with which the model chooses fact over foil using original representations.Erased Acc.is the accuracy after erasure, using our counterfactualq u (x|h ⊥ )distri- bution.Do Acc.measures, for example for Do(C=sg) (see Eq. (11)), the rate at which the intervention induces the model to assign higher probability to thesgelement of the (fact, foil) pair over itsplcounterpart, reported on aggregate oversgandplcontexts in the test set. iments, we noticed that smaller Englishgpt2mod- els forverbal-numberwere also notably worse thangpt2-large. 6.2 Causal Controlled Generation In §4, we argued for a causal structure for lan- guage generation that allows us to intervene on the concept-valued random variableC. We now test this causal model empirically by computing the do- intervention in Eq. (11). We define success for the intervention using the forced-choice setup shown in sentences(1-a)and(1-b). For example, given a context with asgfact, we considerdo(C=pl)suc- cessful ifp(x|H ⊥ =h ⊥ ,do(C=pl))assigns higher probability to theplfoil oversgfact. Results for this experiment are shown in Fig. 3. For context, we report the model’s accuracy in the forced-choice setup before (Orig. Acc.) and after (Erased Acc.) erasure. We note the consistency of results between information-theoretic metrics in Table 2 and post-erasure accuracy in Fig. 3—the erasure intervention successfully lowers the accu- racy of the minority classplforverbal-number, however the intervention fails to significantly re- duce accuracy forgrammatical-gender. With this context in mind, the do-intervention is remarkably successful forverbal-number Binary Overall Sentiment3-way Overall Sentiment5-way Overall Sentiment ModelMethodICaCE- cosine (↓) ICaCE- L2 (↓) ICaCE- normdiff(↓) ICaCE- cosine (↓) ICaCE- L2 (↓) ICaCE- normdiff(↓) ICaCE- cosine (↓) ICaCE- L2 (↓) ICaCE- normdiff(↓) bert-base-uncasedBest CEBaB0.64±0.050.31±0.000.30±0.000.54±0.040.56±0.000.48±0.000.63±0.010.74±0.020.54±0.02 bert-base-uncasedINLP0.79±0.000.52±0.050.52±0.050.70±0.010.58±0.020.55±0.010.60±0.020.80±0.020.72±0.03 bert-base-uncasedDo-INLP0.78±0.010.55±0.010.55±0.010.68±0.010.62±0.040.53±0.020.54±0.010.77±0.020.61±0.03 bert-base-uncasedLEACE0.77±0.010.36±0.010.35±0.010.78±0.030.57±0.010.53±0.010.77±0.030.80±0.020.71±0.02 bert-base-uncasedDo-LEACE0.75±0.010.37±0.010.37±0.010.69±0.020.57±0.010.49±0.010.58±0.020.75±0.020.61±0.03 gpt2Best CEBaB0.58±0.010.29±0.000.29±0.000.50±0.010.52±0.010.42±0.010.59±0.010.60±0.020.40±0.01 gpt2INLP1.00±0.000.49±0.040.44±0.031.00±0.000.65±0.040.52±0.011.00±0.000.72±0.020.58±0.02 gpt2Do-INLP0.98±0.020.30±0.010.30±0.000.99±0.020.53±0.010.51±0.010.99±0.010.68±0.020.66±0.02 gpt2LEACE1.00±0.000.31±0.010.30±0.001.00±0.000.53±0.010.51±0.011.00±0.000.68±0.020.66±0.02 gpt2Do-LEACE0.99±0.060.28±0.000.28±0.000.99±0.040.52±0.010.51±0.010.99±0.020.68±0.020.67±0.02 roberta-baseBest CEBaB0.70±0.030.29±0.010.29±0.010.62±0.020.55±0.010.48±0.000.64±0.010.78±0.010.59±0.01 roberta-baseINLP0.80±0.000.32±0.020.32±0.020.72±0.010.55±0.000.54±0.010.59±0.010.84±0.010.81±0.01 roberta-baseDo-INLP0.79±0.000.30±0.060.30±0.060.70±0.010.52±0.010.47±0.020.56±0.010.80±0.000.72±0.01 roberta-baseLEACE0.82±0.010.31±0.010.31±0.010.83±0.010.54±0.010.51±0.010.83±0.010.83±0.010.80±0.01 roberta-baseDo-LEACE0.77±0.010.31±0.010.31±0.010.70±0.010.53±0.010.47±0.010.59±0.010.81±0.010.76±0.01 Table 3: CEBaB benchmark results for our causal do-intervention. In three sets of columns, we report empirical Individual Causal Concept Effect (ICaCE) losses for 2-, 3- and 5-class overall sentiment prediction. Three types of losses are reported, lower is better for all three.ICaCE-cosineindicates whether the estimated and observed effect have the same direction, without accounting for magnitude.ICaCE-L2compares the Euclidean norm of the difference of the estimated and observed effect, and therefore reflects both magnitude and direction differences. ICaCE-normdiffis the absolute difference between the Euclidean norms of the observed and estimated effects, thereby honing in on magnitude differences. Each entry shows mean±standard deviation over random restarts. We report results for the best performing method from CEBaB (Abraham et al., 2022), INLP (Ravfogel et al., 2020), and LEACE (Belrose et al., 2023). We apply our causal do-intervention from §4.2, this time in a classification setting, usingPreturned by INLP and LEACE. See §6.3 for discussion of these results. across two models.By acting solely in our concept subspace, we are able to almost match the original accuracy for both models. Results forgpt2-base-frenchare much worse, since the do-intervention actually has lower accuracy than erasure. Viewed together with results in §6.1, this confirms that our causal structure only holds given an adequatePunder our counterfactual framework. Nonetheless, the success of the do-intervention on verbal-numberdespite an imperfect partitioning of concept information suggests that identifying the causal concept direction is not a necessary requirement for causal concept-based controlled generation, so long asH ∥ contains a significant share of concept information. 6.3 CEBaB Benchmark Results As explained in §7, our work joins an ongoing conversationaboutwhetherinterpretability methods such as linear concept erasure, which operate on low-level neural representations, are good estimators for a high-level causal narrative, e.g., the process by which a language model generates grammatical text (Geiger et al., 2023). The CEBaB benchmark (Abraham et al., 2022) evaluates concept-based explanation methods as causal effect estimators by comparing the change in a sentiment classifier’s predictions resulting, e.g., from concept erasure, against a ground truth. This ground truth is the change in the classifier’s predictions from changing the concept value in the input text, without modifying other aspects of the restaurant review. In Table 3, we report CEBaB results for INLP (Ravfogel et al., 2020) and LEACE (Belrose et al., 2023), our do-intervention with the projection matrices returned by these methods, and the best performing score from other methods in CEBaB. Results show that our do-intervention improves the performance of INLP and LEACE on CEBaB. For LEACE, our do-intervention improves the directional (cosine) loss, but not the magnitude losses. The inverse is true for INLP. Across models and tasks, our method occasionally beats other methods on CEBaB. 7 Related Work The use of concept subspaces for estimating causal effects and performing causal interventions is a well-studied area. Feder et al. (2021) introduce a method for estimating the causal effect of a concept encoding on a language model’s predictions, in a manner that accounts for confounders. They use adversarial training to obtain counterfactual repre- sentations with a different concept value, without changing the textual context. The CEBaB bench- mark (Abraham et al., 2022) builds on this work by providing a dataset of textual counterfactuals, enabling the comparison of different concept inter- vention methods. Arora et al. (2024) also provide textual counterfactuals to benchmark a much larger set of linguistics tasks. Elazar et al. (2021) and Lasri et al. (2022) measure changes in model accu- racy on a concept-related task as a test of whether a model uses a linear concept encoding. Jacovi et al. (2021), Ravfogel et al. (2021), Park et al. (2023), Geiger et al. (2024), and Wu et al. (2024) create concept counterfactual representations using linear projection. Geiger et al. (2021, 2023) propose a the- oretical framework relating low-level interpretabil- ity methods such as linear concept erasure to high level causal phenomena such as text generation. Previous work in linear concept erasure is also interested in measuring the degree to which erasure preserves non-concept-related features. Ravfogel et al. (2020, 2022a,b) perform various tests, e.g., evaluating whether the semantics of the representation space were affected by erasure using SimLex-999 (Hill et al., 2015), which is different from whether the language model’s predictions have changed. Elazar et al. (2021) assess damage top LM via two tests: First, they test the model’s ability to recover task performance after finetuning, and second, they report the overall KL divergence in the LM’s output distribution, over the entire vocabulary. This last approach was a source of inspiration for our work, which delves much deeper into this distributional distance idea via our stability and containment tests. 8 Conclusion In this paper, we set out to define anintrinsic measure of information in a subspace of a language model’s representation space.In light of the correlational failure mode of linear concept erasure methods (Kumar et al., 2022), doing so requires a counterfactual approach: By assuming statistical independence between the components of a representation in the concept subspace and its orthogonal complement, we are able to correctly measure information in a subspace by marginalizing out the remainder of the space. To the extent that a causal concept subspace exists for a particular concept and model, erasure under this metric is optimized by that subspace. In practice, we did not actually optimize this metric, because it is computationally intractable due to nested sums over the infinite representations spaceH. We leave the development of a tractable approximation to fu- ture work. Our theoretical analysis, combined with the efficacy of linear erasure methods using a cor- relational objective, suggests a tantalizing prospect: That a counterfactual objective could identify a one-dimensional causal subspace containingall information about the concept empirically. References Eldar David Abraham, Karel D’Oosterlinck, Amir Feder, Yair Ori Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. 2022. CEBaB: Estimating the causal effects of real- world concepts on NLP model behavior. InAd- vances in Neural Information Processing Sys- tems, volume 35, pages 17582–17596. Curran Associates, Inc. Afra Amini, Tiago Pimentel, Clara Meister, and Ryan Cotterell. 2023. Naturalistic causal prob- ing for morpho-syntax.Transactions of the As- sociation for Computational Linguistics, 11:384– 403. Aryaman Arora, Dan Jurafsky, and Christopher Potts. 2024. CausalGym: Benchmarking causal interpretability methods on linguistic tasks. arXiv:2402.12560. Matthew Baerman. 2007. Syncretism.Language and Linguistics Compass, 1(5):539–551. Nora Belrose, David Schneider-Joseph, Shauli Rav- fogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. LEACE: Perfect linear con- cept erasure in closed form.arXiv preprint arXiv:2306.03819. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. InAdvances in Neural Information Processing Systems, volume 29. Curran Associates, Inc. Cristina Bosco and Manuela Sanguinetti. 2014. To- wards a Universal Stanford Dependencies paral- lel treebank. InProceedings of the 13th Work- shop on Treebanks and Linguistic Theories (TLT- 13), pages 14–25, Tubingen, Germany. Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Ben- gio. 2016. Generating sentences from a continu- ous space. InProceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 10–21, Berlin, Germany. Asso- ciation for Computational Linguistics. Thomas M. Cover and Joy M. Thomas. 2006.El- ements of Information Theory, second edition. Wiley-Interscience. Li Du, Lucas Torroba Hennigen, Tiago Pimentel, Clara Meister, Jason Eisner, and Ryan Cotterell. 2023. A measure-theoretic characterization of tight language models. InProceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 9744–9770, Toronto, Canada. Association for Computational Linguistics. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behav- ioral explanation with amnesic counterfactuals. Transactions of the Association for Computa- tional Linguistics, 9:160–175. Amir Feder, Nadav Oved, Uri Shalit, and Roi Re- ichart. 2021. CausaLM: Causal model expla- nation through counterfactual language models. Computational Linguistics, 47(2):333–386. Wikimedia Foundation. 2023. Wikimedia down- loads. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. InAdvances in Neural Infor- mation Processing Systems, volume 34, pages 9574–9586. Curran Associates, Inc. Atticus Geiger, Chris Potts, and Thomas Icard. 2023. Causal abstraction for faithful model in- terpretation. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. 2024. Find- ing alignments between interpretable causal vari- ables and distributed neural representations. In Proceedings of the Third Conference on Causal Learning and Reasoning, volume 236 ofPro- ceedings of Machine Learning Research, pages 160–187. PMLR. Yoav Goldberg. 2019. Assessing BERT’s syntactic abilities.ArXiv. Bruno Guillaume, Marie-Catherine de Marneffe, and Guy Perrier. 2019. Conversion et améliora- tions de corpus du français annotés en Universal Dependencies.Revue TAL, 60(2):71–95. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation.Computational Linguistics, 41(4):665–695. Alon Jacovi, Swabha Swayamdipta, Shauli Ravfo- gel, Yanai Elazar, Yejin Choi, and Yoav Gold- berg. 2021. Contrastive explanations for model interpretability. InProceedings of the 2021 Con- ference on Empirical Methods in Natural Lan- guage Processing, pages 1597–1611, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Abhinav Kumar, Chenhao Tan, and Amit Sharma. 2022. Probing classifiers are unreliable for con- cept removal and detection. InAdvances in Neu- ral Information Processing Systems, volume 35, pages 17994–18008. Curran Associates, Inc. Anne Lacheret, Sylvain Kahane, Julie Beliao, Anne Dister, Kim Gerdes, Jean-Philippe Goldman, Nicolas Obin, Paola Pietrandrea, and Atanas Tchobanov. 2014.Rhapsodie: a prosodic- syntactic treebank for spoken French. InPro- ceedings of the Ninth International Confer- ence on Language Resources and Evaluation (LREC’14), pages 295–301, Reykjavik, Ice- land. European Language Resources Association (ELRA). Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. 2022. Probing for the usage of grammatical number. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 8818–8831, Dublin, Ireland. Association for Computational Linguis- tics. Tal Linzen, Emmanuel Dupoux, and Yoav Gold- berg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies.Trans- actions of the Association for Computational Linguistics, 4:521–535. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. InPro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. Ines Montani, Matthew Honnibal, Matthew Honni- bal, Sofie Van Landeghem, Adriane Boyd, Hen- ning Peters, Paul O’Leary McCann, Maxim Sam- sonov, Jim Geovedi, Jim O’Regan, Duygu Al- tinok, György Orosz, Søren Lind Kristiansen, , Roman, Explosion Bot, Lj Miranda, Lean- der Fiedler, Daniël De Kok, Grégory Howard, , Edward, Wannaphong Phatthiyaphaibun, Yohei Tamura, Sam Bozek, , Murat, Mark Amery, Ryn Daniels, Björn Böing, Pradeep Kumar Tippa, and Peter Baumgartner. 2022. spaCy: Industrial- strength natural language processing in Python. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Haji ˇ c, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Ty- ers, and Daniel Zeman. 2020. Universal Depen- dencies v2: An evergrowing multilingual tree- bank collection. InProceedings of the 12th Lan- guage Resources and Evaluation Conference, pages 4034–4043, Marseille, France. European Language Resources Association. OpenAI. 2022. Introducing ChatGPT. Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the ge- ometry of large language models.arXiv preprint arXiv:2311.03658. Judea Pearl. 2009. Causal inference in statistics: An overview.Statistics Surveys, 3:96–146. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Lan- guage models are unsupervised multitask learn- ers. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. InProceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 7237–7256, Online. Association for Computational Linguistics. Shauli Ravfogel, Yoav Goldberg, and Ryan Cot- terell. 2023. Log-linear guardedness and its im- plications. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9413–9431, Toronto, Canada. Association for Computational Linguistics. Shauli Ravfogel, Grusha Prasad, Tal Linzen, and Yoav Goldberg. 2021. Counterfactual interven- tions reveal the causal effect of relative clause representations on agreement prediction. InPro- ceedings of the 25th Conference on Computa- tional Natural Language Learning, pages 194– 209, Online. Association for Computational Lin- guistics. Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. 2022a. Linear adversarial concept erasure. InProceedings of the 39th In- ternational Conference on Machine Learning, volume 162 ofProceedings of Machine Learn- ing Research, pages 18400–18421. PMLR. Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. 2022b. Kernelized concept erasure. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing. Association for Computational Linguis- tics. Manuela Sanguinetti and Cristina Bosco. 2014. Converting the parallel treebank ParTUT in Uni- versal Stanford Dependencies. InProceedings of the 1rst Conference for Italian Computational Linguistics (CLiC-it 2014), Pisa, Italy. Manuela Sanguinetti and Cristina Bosco. 2015. PartTUT: The Turin University Parallel Tree- bank, chapter 1. Springer International Publish- ing, Cham. Claude E. Shannon. 1948. A mathematical theory of communication.The Bell System Technical Journal, 27(3):379–423. Hugo Touvron, Louis Martin, Kevin Stone, Pe- ter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernan- des, Jeremy Fu, Wenyin Fu, Brian Fuller, Cyn- thia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Ko- renev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Ad- ina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:2307.09288. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexan- der Rush. 2020. Transformers: State-of-the-art natural language processing. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pages 38–45, Online. Association for Computational Linguistics. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. 2024. ReFT: Representation finetuning for language models. arXiv:2404.03592. Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. 2020. A theory of usable information under computational con- straints. InInternational Conference on Learn- ing Representations. Kevin Yang and Dan Klein. 2021. FUDGE: Con- trolled text generation with future discriminators. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3511–3535, Online. Associ- ation for Computational Linguistics. A DecomposingI q (C;H) Proposition A.1.SupposePis aε-eraser and(I d −P)is aε-encapsulator ofCwith respect toH. Then, asε→0, the following holds I q (C;H) = I q (C;H ⊥ ) + I q (C;H ∥ )(14) Proof.On the left-hand side, I q (C;H) +ε≥I q (C;H ∥ ) +ε(data-processing inequality)(15a) ≥I q (C;H ∥ ) + I q (C;H ⊥ )(Pis anε-eraser)(15b) On the right-hand side, I q (C;H) +ε≤I q (C;H ∥ ) + 2ε((I d −P)is anε-encapsulator)(16a) ≤I q (C;H ∥ ) + I q (C;H ⊥ ) + 2ε(non-negativity of MI)(16b) Combining Eq. (15b) and Eq. (16b), we have I q (C;H ∥ ) + I q (C;H ⊥ )≤I q (C;H) +ε(17a) ≤I q (C;H ∥ ) + I q (C;H ⊥ ) + 2ε(17b) Takingε→0in Eq. (17), we have Eq. (14) I q (C;H) = I q (C;H ∥ ) + I q (C;H ⊥ ) ■ B Proof of Theorem 4.1 Theorem 4.1.Consider a joint distributionpthat factors as in Fig. 2b, parameterized by orthogonal projection matrixP. Under the distribution p do (x,h ⊥ ,h ∥ ,c) =p(x|h ⊥ ,h ∥ )(12) p(h ⊥ |do (C=c))p(h ∥ |do (C=c))p(c) we have thatPis anε-eraser,I d −Pis anε-encapsulator,I d −Pis anε-container andPis an ε-stabilizer for everyε >0. Proof.Given the factorization in Fig. 2b, we derive the following equation using the independence assumptions given in Fig. 2b: p do (x,h ⊥ ,h ∥ ,c) =p(x|h ⊥ ,h ∥ )p(h ⊥ |do (C=c))p do (h ∥ |do (C=c))p(c)(19a) =p(x|h ⊥ ,h ∥ )p(h ⊥ )p do (h ∥ |c)p(c)(19b) Erasure.Given Eq. (19b), we have the following joint distribution p do (c,h ⊥ ) = X h ∥ ∈H ∥ X x∈Σ p do (x,h ⊥ ,h ∥ ,c)(20a) = X h ∥ ∈H ∥ X x∈Σ p(x|h ⊥ ,h ∥ )p(h ⊥ )p do (h ∥ |c)p(c)(20b) = X h ∥ ∈H ∥ X x∈Σ p(x|h ⊥ ,h ∥ ) ! | z =1 p(h ⊥ )p do (h ∥ |c)p(c)(20c) = X h ∥ ∈H ∥ p do (h ∥ |c) | z =1 p(h ⊥ )p(c)(20d) =p(h ⊥ )p(c)(20e) The mutual informationI(C;H ⊥ )can be computed as follows I(C;H ⊥ ) = X c∈C X h ⊥ ∈H ⊥ p do (c,h ⊥ ) log p do (c,h ⊥ ) p(c)p(h ⊥ ) (21a) = X c∈C X h ⊥ ∈H ⊥ p do (c,h ⊥ ) log p(h ⊥ )p(c) p(c)p(h ⊥ ) (applying Eq. (20e))(21b) = 0< ε(21c) for everyε >0. Encapsulation.The following equation holds given Eq. (19b) I(C;H)−I(C;H ∥ ) = I(C;H ∥ ,H ⊥ )−I(C;H ∥ )(H=H ⊥ ,H ∥ )(22a) = I(C;H ⊥ |H ∥ )(22b) = I(C;H ⊥ )(H ⊥ ,H ∥ are independent (§3.3))(22c) = 0< ε(applying Eq. (21c))(22d) (22e) Containment.The following joint distribution can be derived from Eq. (19b) p do (x,h ∥ ,c=c) = X h ⊥ ∈H ⊥ p do (x,h ⊥ ,h ∥ ,c=c)(23a) = X h ⊥ ∈H ⊥ p(x|h ⊥ ,h ∥ )p(h ⊥ )p do (h ∥ |c=c)p(c=c)(23b) = X h ⊥ ∈H ⊥ p(x,h ⊥ ,h ∥ ) p(h ⊥ ,h ∥ ) p(h ⊥ )p do (h ∥ |c=c)p(c=c)(23c) = X h ⊥ ∈H ⊥ p(x,h ⊥ ,h ∥ ) p(h ⊥ )p(h ∥ ) p(h ⊥ )p do (h ∥ |c=c)p(c=c)(H ⊥ ,H ∥ are independent (§3.3)) (23d) = X h ⊥ ∈H ⊥ p(x,h ⊥ |h ∥ ) |z =p(x|h ∥ ) p do (h ∥ |c=c)p(c=c)(23e) =p(x|h ∥ )p do (h ∥ |c=c)p(c=c)(23f) =p(x|h ∥ ,c=c)p(h ∥ |c=c)(H ∥ is deterministic givenC) (23g) The mutual informationI(X;H ∥ |C=c)can be computed as follows I(X;H ∥ |C=c)(24a) = X x∈Σ X h ∥ ∈H ∥ p do (x,h ∥ ,c=c) log p do (x,h ∥ ,c=c) p(x|c=c)p(h ∥ |c=c) (24b) = X x∈Σ X h ∥ ∈H ∥ p do (x,h ∥ ,c=c) log p(x|h ∥ ,c=c)p(h ∥ |c=c) p(x|c=c)p(h ∥ |c=c) (applying Eq. (23g)) (24c) = X x∈Σ X h ∥ ∈H ∥ p do (x,h ∥ ,c=c) log p(x|h ∥ ,c=c)p(h ∥ |c=c) p(x|h ∥ ,c=c)p(h ∥ |c=c) (H ∥ is deterministic givenC) (24d) = 0< ε(24e) Stability.The following equation holds given Eq. (19b) I(X;H|C=c)−I(X;H ⊥ |C=c)(25a) =I(X;H ⊥ ,H ∥ |C=c)−I(X;H ⊥ |C=c)(H= (H ⊥ ,H ∥ ))(25b) =I(X;H ∥ |H ⊥ ,C=c)(conditional mutual information)(25c) =I(X;H ∥ |C=c)(H ⊥ ,H ∥ are independent (§3.3))(25d) =0< ε(applying Eq. (24e))(25e) ■ C Concept Word Lists and CEBaB Prompts Ambiance The ambiance was The atmosphere was The restaurant was The vibe was The setting was Food The cuisine was The dishes were The meal was The food was The flavors was Noise The ambient noise level was The background noise was The surrounding sound was The auditory atmosphere was The ambient soundscape was Service The service was The staff was The hospitality extended by the staff was The waiter was The host was Table 4: List of prompts for CEBaB dataset. verbal-number sgabsorbs, accepts, accompanies, accounts, achieves, acknowledges, acti- vates, adds, addresses, administers, admits, adopts, advises, advocates, affects, agrees, aims, allows, announces, appears, applies, appoints, . . . plabsorb, accept, accompany, account, achieve, acknowledge, activate, add, address, administer, admit, adopt, advise, advocate, affect, agree, aim, allow, announce, appear, apply, appoint, . . . grammatical-gender mscabbatial, absolu, actif, actuel, additionnel, administratif, afro-américain, agressif, aigu, algérien, allemand, alsacien, amer, américain, ancien, annuel, architectural, arménien, artificiel, artisanal, . . . femabbatiale, absolue, active, actuelle, additionnelle, administrative, afro- américaine, agressive, aiguë, algérienne, allemande, alsacienne, amère, américaine, ancienne, annuelle, architecturale, arménienne, artificielle, artisanale, . . . Table 5: Subset of linguistic concept word lists. ambiance poscozy, elegant, inviting, charming, welcoming, intimate, sophisticated, tranquil, lively, romantic, chic, rustic, vibrant, serene, stylish, eclectic, enchanting, upscale, warm, bustling, idyllic, exquisite, radiant, harmonious, blissful, alluring, picturesque, opulent, sumptuous, dreamy, luxurious, polished, effervescent, enthralling negdingy, claustrophobic, dreary, uninviting, dull, sterile, disorganized, loud, cramped, stale, unpleasant, chaotic, gaudy, tacky, uncomfortable, grimy, stuffy, depressing, drab, cold, seedy, pretentious, overcrowded, gloomy, oppressive, grim, tense, repellent, muggy, sullen, bland, repugnant, dismal, shabby food posdelicious, mouthwatering, flavorful, delectable, savory, scrumptious, tasty, heavenly, exquisite, succulent, aromatic, satisfying, appetizing, divine, gourmet, nutritious, fresh, yummy, fragrant, sumptuous, delightful, rich, zesty, indulgent, juicy, decadent, bal- anced, sweet, tangy, refreshing, warm, tender, crispy, herbacious, spiced negtasteless, bland, overcooked, undercooked, stale, soggy, greasy, unappetizing, flavorless, dry, tough, burnt, rancid, unpalatable, watery, disgusting, sour, bitter, mushy, unpleasant, unappealing, foul, off-putting, insipid, rubbery, dull, moldy, spoiled, repulsive, stinky, unbalanced, salty, fatty, stringy, oily noise posvibrant, lively, energetic, buoyant, festive, animated, cheerful, convivial, invigorating, buzzy, jubilant, pulsating, tranquil, serene, calm, peaceful, quiet, relaxed, soothing, gentle, mellow, harmonious, rejuvenating, vivacious, resonant negdisruptive, deafening, chaotic, clamorous, unruly, boisterous, raucous, overwhelming, harsh, grating, jarring, unpleasant, discordant, intrusive, irritating, nerve-wracking, agitated, distracting, unbearable, silent, loud, obnoxious, booming, cacaphonous, blaring service posattentive, friendly, efficient, professional, courteous, prompt, welcoming, accommo- dating, hospitable, personable, polished, gracious, knowledgeable, warm, engaging, diligent, exemplary, seamless, outstanding, enthusiastic, meticulous, personalized, con- siderate, anticipatory, respectful, reliable, consistent, thoughtful, empathetic, genuine, polite, proactive, adaptable, detail-oriented, impeccable neginattentive, slow, rude, incompetent, dismissive, disorganized, impersonal, unprofes- sional, neglectful, indifferent, abrupt, inefficient, aloof, uncaring, clueless, forgetful, disrespectful, aggressive, insubordinate, inept, unfriendly, unaccommodating, inconsid- erate, arrogant, sloppy, unknowledgeable, intrusive, negligent, careless, unresponsive, unreliable, distracted, impolite, disinterested, discourteous Table 6: Full list of CEBaB concept words.