Paper deep dive
The Geometry of Categorical and Hierarchical Concepts in Large Language Models
Kiho Park, Yo Joong Choe, Yibo Jiang, Victor Veitch
Models: Gemma-2B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 8:02:12 PM
Summary
The paper formalizes the 'linear representation hypothesis' for LLMs by extending binary concept representations to vector-based representations of categorical concepts. It demonstrates that categorical concepts can be represented as polytopes in representation space and proves that hierarchical relationships between concepts are encoded geometrically as orthogonality between subspaces. These findings are validated using Gemma and LLaMA-3 models with data from WordNet.
Entities (5)
Relation Signals (3)
Gemma → validatedby → Linear Representation Hypothesis
confidence 95% · We validate these theoretical results on the Gemma and LLaMA-3 large language models
Llama-3 → validatedby → Linear Representation Hypothesis
confidence 95% · We validate these theoretical results on the Gemma and LLaMA-3 large language models
WordNet → providesdatafor → Linear Representation Hypothesis
confidence 90% · estimating representations for 900+ hierarchically related concepts using data from WordNet
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The linear representation hypothesis is the informal idea that semantic concepts are encoded as linear directions in the representation spaces of large language models (LLMs). Previous work has shown how to make this notion precise for representing binary concepts that have natural contrasts (e.g., {male, female}) as directions in representation space. However, many natural concepts do not have natural contrasts (e.g., whether the output is about an animal). In this work, we show how to extend the formalization of the linear representation hypothesis to represent features (e.g., is_animal) as vectors. This allows us to immediately formalize the representation of categorical concepts as polytopes in the representation space. Further, we use the formalization to prove a relationship between the hierarchical structure of concepts and the geometry of their representations. We validate these theoretical results on the Gemma and LLaMA-3 large language models, estimating representations for 900+ hierarchically related concepts using data from WordNet.
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
136,082 characters extracted from source content.
Expand or collapse full text
The Geometry of Categorical and Hierarchical Concepts in Large Language Models Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch University of Chicago Abstract The linear representation hypothesis is the informal idea that semantic concepts are encoded as linear directions in the representation spaces of large language models (LLMs). Previous work has shown how to make this notion precise for representing binary concepts that have natural contrasts (e.g., male,femalemalefemale\ male, female\ male , female ) as directions in representation space. However, many natural concepts do not have natural contrasts (e.g., whether the output is about an animal). In this work, we show how to extend the formalization of the linear representation hypothesis to represent features (e.g., is_animal) as vectors. This allows us to immediately formalize the representation of categorical concepts as polytopes in the representation space. Further, we use the formalization to prove a relationship between the hierarchical structure of concepts and the geometry of their representations. We validate these theoretical results on the Gemma and LLaMA-3 large language models, estimating representations for 900+ hierarchically related concepts using data from WordNet.111Code is available at github.com/KihoPark/LLM_Categorical_Hierarchical_Representations. 1 Introduction Understanding how high-level semantic meaning is encoded in the representation spaces of large language models (LLMs) is a fundamental problem in interpretability. A particularly promising avenue is the linear representation hypothesis (e.g., Mikolov et al., 2013; Elhage et al., 2022; Nanda et al., 2023; Gurnee & Tegmark, 2024; Park et al., 2024). This is the informal hypothesis that semantic concepts are represented linearly in the representation spaces of LLMs. To assess the validity of this hypothesis, and systematically build tools on top of it, we must make precise what it means for a concept to be linearly represented, and understand how semantics are encoded in (the geometry of) the representation spaces. Focusing on the final softmax layer, Park et al. (2024) give a formalization for the case of binary concepts that can be defined by counterfactual pairs of words. For example, the concept male⇒female⇒malefemale male femalemale ⇒ female is formalized using the counterfactual pairs (“man”,“woman”),\(``man′,``woman′), ( “man” , “woman” ) , (“king”,“queen”),…(``king′,``queen′),…\( “king” , “queen” ) , … . They prove that such binary concepts have a well-defined linear representation as a direction in the representation space. They further connect semantic structure and representation geometry by showing that, under a suitable inner product, causally separable concepts that can be freely manipulated (e.g., male⇒female⇒malefemale male femalemale ⇒ female and french⇒english⇒frenchenglish french englishfrench ⇒ english) are represented by orthogonal directions. However, this formalization is limited: many natural concepts cannot be defined by counterfactual pairs of words. For example, simple binary features (e.g., is_animal) or categorical concepts (e.g., mammal,bird,reptile,fishmammalbirdreptilefish\ mammal, bird, reptile, fish\ mammal , bird , reptile , fish ) do not admit such formalizations. Additionally, it is not clear how semantic relationships beyond causal separability are encoded in the representation space. In particular, we are interested in this paper in understanding how hierarchical relationships between concepts are encoded in the representation space. That is, what is the relationship between the representations of animaltypewriter_animal, mammaltypewriter_mammal, and dogtypewriter_dog? In this paper, we extend the linear representation hypothesis as follows: 1. We show how to move from representations of binary concepts as directions to representations as vectors. As a straightforward consequence, this allows us to represent categorical concepts (e.g., mammal,bird,reptile,fishmammalbirdreptilefish\ mammal, bird, reptile, fish\ mammal , bird , reptile , fish ) as polytopes where each vertex is the vector representation of one of the elements of the concept (e.g., is_bird). 2. Using this result, we show that semantic hierarchy between concepts is encoded geometrically as orthogonality between representations, in a manner we make precise. 3. Finally, we empirically validate these theoretical results on the Gemma (Mesnard et al., 2024) and LLaMA-3 (Dubey et al., 2024) LLMs. To that end, we extract concepts from the WordNet hierarchy (Miller, 1995), estimate their representations, and show that the geometric structure of the representations aligns with the semantic hierarchy of WordNet. The final structure is remarkably simple, and is summarized in Figure 1. Figure 1: In the representation spaces of LLMs, hierarchically related concepts (such as plant⇒animal⇒plantanimal plant animalplant ⇒ animal and mammal⇒bird⇒mammalbird mammal birdmammal ⇒ bird) live in orthogonal subspaces, while categorical concepts are represented as polytopes. The top panel illustrates the structure; the bottom panels show the measured representation structure in the Gemma LLM. See Section 5 and Appendix A for details. 2 Preliminaries We begin with some necessary background. 2.1 Large Language Models For the purposes of this paper, we consider a large language model to consist of two parts. The first part is a function λ that maps an input text x to a vector λ(x)λ(x)λ ( x ) in a representation space Λ≃ℝdsimilar-to-or-equalsΛsuperscriptℝ ^dΛ ≃ blackboard_Rd. This is the function given by the stacked transformer blocks. We take λ(x)λ(x)λ ( x ) to be the output of the final layer at the final token position. The second part is an unembedding layer that assigns a vector γ(y)γ(y)γ ( y ) in an unembedding space Γ≃ℝdsimilar-to-or-equalsΓsuperscriptℝ ^dΓ ≃ blackboard_Rd to each token y in the vocabulary. Together, these define a sampling distribution over tokens via the softmax distribution: ℙ(y|x)=exp(λ(x)⊤γ(y))∑y′∈Vocabexp(λ(x)⊤γ(y′)).ℙconditionalsuperscripttopsubscriptsuperscript′Vocabsuperscripttopsuperscript′P(y~|~x)= (λ(x) γ(y)) _y % (λ(x) γ(y )).blackboard_P ( y | x ) = divide start_ARG exp ( λ ( x )⊤ γ ( y ) ) end_ARG start_ARG ∑y′ ∈ Vocab exp ( λ ( x )⊤ γ ( y′ ) ) end_ARG . (2.1) We note that the results that follow rely on the duality between the embedding and unembedding spaces, and the softmax link between them. Accordingly, we do not address the “internal” structure of the LLMs. However, we are optimistic that a clear understanding of the softmax geometry will shed light on this as well. 2.2 Concepts We formalize a concept as a latent variable W that is caused by the context X and causes the output Y. That is, a concept is a thing that could—in principle—be manipulated to affect the output of the language model. In the particular case where a concept is a binary variable with a word-level counterfactual, we can identify the variable W with the counterfactual pair of outputs (Y(0),Y(1))01(Y(0),Y(1))( Y ( 0 ) , Y ( 1 ) ). Concretely, we can identify male⇒female⇒malefemale male femalemale ⇒ female with (Y(0),Y(1))∈R(“man”,“woman”),(Y(0),Y(1)) _R\(``man′,``woman′),( Y ( 0 ) , Y ( 1 ) ) ∈R ( “man” , “woman” ) , (“king”,“queen”),(“he”,“her”),…(``king′,``queen′),(``he′,``her′),…\( “king” , “queen” ) , ( “he” , “her” ) , … . We emphasize that the notion of a concept as a latent variable that affects the output is more general than the counterfactual binary case. Given a pair of concept variables W and Z, we say that W is causally separable with Z if the potential outcome Y(W=w,Z=z)formulae-sequenceY(W=w,Z=z)Y ( W = w , Z = z ) is well-defined for all w,zw,zw , z. That is, two variables are causally separable if they can be freely manipulated—e.g., we can change the output language and the sex of the subject freely, so these concepts are causally separable. 2.3 Causal Inner Product and Linear Representations We are trying to understand how concepts are represented. At this stage, there are two distinct representation spaces: Λ Λ and Γ Γ. The former is the space of context embeddings, and the latter is the space of token unembeddings. We would like to unify these spaces so that there is just a single notion of representation. Park et al. (2024) show how to achieve this unification via a “Causal Inner Product”. This is a particular choice of inner product that respects the semantics of language in the sense that the linear representations of (binary, counterfactual) causally separable concepts are orthogonal under the inner product. Their result can be understood as saying that there is some invertible matrix A and constant vector γ¯0subscript¯0 γ_0over¯ start_ARG γ end_ARG0 such that, if we transform the embedding and unembedding spaces as g(y)←A(γ(y)−γ¯0),ℓ(x)←A−⊤λ(x)formulae-sequence←subscript¯0←ℓsuperscriptabsenttopg(y)← A(γ(y)- γ_0), (x)← A^- % λ(x)g ( y ) ← A ( γ ( y ) - over¯ start_ARG γ end_ARG0 ) , ℓ ( x ) ← A- ⊤ λ ( x ) (2.2) then the Euclidean inner product in the transformed spaces is the causal inner product, and the Riesz isomorphism between the embedding and unembedding spaces is simply the usual vector transpose operation. We can estimate A as the whitening operation for the unembedding matrix. Following this transformation, we can think of the embedding and unembedding spaces as the same space, equipped with the Euclidean inner product.222We are glossing over some technical details here; see Park et al. (2024) for details. Notice that the softmax probabilities (eq. 2.1) are unchanged for any A and γ¯0subscript¯0 γ_0over¯ start_ARG γ end_ARG0, so this transformation does not affect the model’s behavior. The vector γ¯0subscript¯0 γ_0over¯ start_ARG γ end_ARG0 defines an origin for the unembedding space, and can be chosen arbitrarily. We give a particularly convenient choice below. In this unified space, the linear representation of a binary concept W∈R0,1subscript01W _R\0,1\W ∈R 0 , 1 is defined as: Definition 1. A vector ℓ¯Wsubscript¯ℓ _Wover¯ start_ARG ℓ end_ARGW is a linear representation of a binary concept W if for all contexts ℓ ℓ, and all concept variables Z that are causally separable with W, we have, for all α>00α>0α > 0, ℙ(W=1|ℓ+αℓ¯W)ℙconditional1ℓsubscript¯ℓ (W=1~|~ +α _W)blackboard_P ( W = 1 | ℓ + α over¯ start_ARG ℓ end_ARGW ) >ℙ(W=1|ℓ), andabsentℙconditional1ℓ and >P(W=1~|~ ), and> blackboard_P ( W = 1 | ℓ ) , and (2.3) ℙ(Z|ℓ+αℓ¯W)ℙconditionalℓsubscript¯ℓ (Z~|~ +α _W)blackboard_P ( Z | ℓ + α over¯ start_ARG ℓ end_ARGW ) =ℙ(Z|ℓ).absentℙconditionalℓ =P(Z~|~ ).= blackboard_P ( Z | ℓ ) . (2.4) That is, the linear representation is a direction in the representation space that, when added to the context, increases the probability of the concept, but does not affect the probability of any off-target concept. The representation is merely a direction because αℓ¯Wsubscript¯ℓα _Wα over¯ start_ARG ℓ end_ARGW is also a linear representation for any α>00α>0α > 0 (i.e., there is no notion of magnitude). In the case of concepts corresponding to counterfactual pairs of words, this direction can be shown to be proportional to the “linear probing” direction, and proportional to g(Y(1))−g(Y(0))10g(Y(1))-g(Y(0))g ( Y ( 1 ) ) - g ( Y ( 0 ) ) for any counterfactual pair Y(1),Y(0)10Y(1),Y(0)Y ( 1 ) , Y ( 0 ) that differ on W. 3 General Concepts and Hierarchical Structure Our high-level strategy will be to build up from binary concepts to more complex structure. We begin by defining the basic building blocks. Binary and Categorical Concepts The most general concept we address in this paper is a categorical concept, which refers to any concept corresponding to a categorical latent variable. This includes binary concepts as a special case. We consider two kinds of binary concept: binary features and binary contrasts. A binary feature W∈Rnot_w,is_wsubscriptnot_wis_wW _R\ not\_w, is\_w\W ∈R not_w , is_w is an indicator of whether the output has the attribute w. For example, if the feature __ is\_animaltypewriter_is _ typewriter_animal is true, then the output will be about an animal. A binary contrast a⇒b∈Ra,b⇒absubscript a b _R\a,b\a ⇒ b ∈R a , b is a binary variable that contrasts two specific attribute values. For example, the variable mammal⇒bird⇒mammalbird mammal birdmammal ⇒ bird is a binary contrast. In the particular case where the binary contrast can correspond to counterfactual pairs of words (e.g., male⇒female⇒malefemale male femalemale ⇒ female), the concept matches the definition used in Park et al. (2024). Hierarchical Structure The next step is to define what we mean by a hierarchical relation between concepts. To that end, to each attribute w, we associate a set of tokens (w)Y(w)Y ( w ) that have the attribute. For example, (mammal)=“ dog”,“ cats”,“ Tiger”,…mammal“ dog”“ cats”“ Tiger”…Y( mammal)=\`` dog′,`` cats′,`` Tiger% ′,…\Y ( mammal ) = “ dog” , “ cats” , “ Tiger” , … . Then, Definition 2. A value z is subordinate to a value w (denoted by z≺wprecedesz wz ≺ w) if (z)⊆(w)Y(z) (w)Y ( z ) ⊆ Y ( w ). We say a categorical concept Z∈Rz0,…,zn−1subscriptsubscript0…subscript1Z _R\z_0,…,z_n-1\Z ∈R z0 , … , zitalic_n - 1 is subordinate to a categorical concept W∈Rw0,…,wm−1subscriptsubscript0…subscript1W _R\w_0,…,w_m-1\W ∈R w0 , … , witalic_m - 1 if there exists a value wZsubscriptw_Zwitalic_Z of W such that each value zisubscriptz_izitalic_i of Z is subordinate to wZsubscriptw_Zwitalic_Z. For example, the binary contrast dog⇒cat⇒dogcat dog catdog ⇒ cat is subordinate to the binary feature is_mammal,\ is\_mammal, is_mammal , not_mammal not\_mammal\not_mammal , and the binary contrast parrot⇒eagle⇒parroteagle parrot eagleparrot ⇒ eagle is subordinate to the categorical concept mammal,\ mammal, mammal , bird,bird bird,bird , fish fish\fish . On the other hand, dog⇒eagle⇒dogeagle dog eagledog ⇒ eagle is not subordinate to bird⇒mammal⇒birdmammal bird mammalbird ⇒ mammal, and bird⇒mammal⇒birdmammal bird mammalbird ⇒ mammal and live_in_house⇒live_in_water⇒live_in_houselive_in_water live\_in\_house live\_in\_waterlive_in_house ⇒ live_in_water are not subordinate to each other. Linear Representations of Binary Concepts Now we return to the question of how binary concepts are represented. A key desideratum is that if ℓ¯Wsubscript¯ℓ _Wover¯ start_ARG ℓ end_ARGW is a linear representation, then moving the context embedding in this direction should modify the probability of the target concept in isolation. If adding ℓ¯Wsubscript¯ℓ _Wover¯ start_ARG ℓ end_ARGW also modified off-target concepts, it would not be natural to identify it with the target concept W. In Definition 1, this idea is formalized by the requirement that the probability of causally separable concepts is unchanged when the representation is added to the context. We now observe that, when there is hierarchical structure, this requirement is not strong enough to capture ‘off-target’ behavior. For example, if ℓ¯animalsubscript¯ℓanimal _ animalover¯ start_ARG ℓ end_ARGanimal captures the concept of animal vs not-animal, then moving in this direction should not affect the relative probability of the output being about a mammal versus a bird. If it did, then the representation would actually capture some amalgamation of the animal and mammal concepts. Accordingly, we must strengthen our definition: Definition 3. A vector ℓ¯Wsubscript¯ℓ _Wover¯ start_ARG ℓ end_ARGW is a linear representation of a binary concept W if ℙ(W=1|ℓ+αℓ¯W)ℙconditional1ℓsubscript¯ℓ (W=1~|~ +α _W)blackboard_P ( W = 1 | ℓ + α over¯ start_ARG ℓ end_ARGW ) >ℙ(W=1|ℓ), andabsentℙconditional1ℓ, and >P(W=1~|~ ), and> blackboard_P ( W = 1 | ℓ ) , and (3.1) ℙ(Z|ℓ+αℓ¯W)ℙconditionalℓsubscript¯ℓ (Z~|~ +α _W)blackboard_P ( Z | ℓ + α over¯ start_ARG ℓ end_ARGW ) =ℙ(Z|ℓ),absentℙconditionalℓ =P(Z~|~ ),= blackboard_P ( Z | ℓ ) , (3.2) for all contexts ℓ ℓ, all α>00α>0α > 0, and all concept variables Z that are either subordinate to or causally separable with W. Here, if W is a binary feature for an attribute w, then W=11W=1W = 1 denotes W=is_wis_wW= is\_wW = is_w. Notice that, in the case of binary contrasts defined by counterfactual pairs, this definition is equivalent to Definition 1, because such variables have no subordinate concepts. 4 Representations of Complex Concepts We now turn to how complex concepts are represented. The high-level strategy is to show how to represent binary features as vectors, show how geometry encodes semantic composition, and then use this to construct representations of complex concepts. 4.1 Vector Representations of Binary and Categorical Concepts To build up to complex concepts, we need to understand how to compose linear representations of binary features. At this stage, the representations are only directions in the representation space—they do not have a natural notion of magnitude. In particular, this means we cannot use vector operations (such as addition) to compose representations. To overcome this, we now show how to associate a magnitude to the linear representation of a binary feature. The key is the following result, which connects binary feature representations and word unembeddings: Theorem 4 (Magnitudes of Linear Representations). Suppose there exists a linear representation (normalized direction) ℓ¯Wsubscript¯ℓ _Wover¯ start_ARG ℓ end_ARGW of a binary feature W for an attribute w. Then, there is a constant bw>0subscript0b_w>0bitalic_w > 0 and a choice of unembedding space origin γ¯0wsuperscriptsubscript¯0 γ_0^wover¯ start_ARG γ end_ARG0w in eq. 2.2 such that ℓ¯W⊤g(y)=bwif y∈(w)ℓ¯W⊤g(y)=0if y∉(w).casessuperscriptsubscript¯ℓtopsubscriptif superscriptsubscript¯ℓtop0if cases _W g(y)=b_w&if y (w)\\ _W g(y)=0&if y (w). cases start_ROW start_CELL over¯ start_ARG ℓ end_ARGW⊤ g ( y ) = bitalic_w end_CELL start_CELL if y ∈ Y ( w ) end_CELL end_ROW start_ROW start_CELL over¯ start_ARG ℓ end_ARGW⊤ g ( y ) = 0 end_CELL start_CELL if y ∉ Y ( w ) . end_CELL end_ROW (4.1) Further, if there exist d attributes w0,…,wd−1subscript0…subscript1\w_0,…,w_d-1\ w0 , … , witalic_d - 1 such that the linear representations of the binary features for these attributes are linearly independent, we can choose a canonical origin γ¯0subscript¯0 γ_0over¯ start_ARG γ end_ARG0 in eq. 2.2. All proofs are given in Appendix B. The theorem says that, if a (perfect) linear representation of the animaltypewriter_animal feature exists, then every token having the animal attribute has the same dot product with the representation vector; i.e., “cat” is exactly as much animaltypewriter_animal as “dog” is. If this weren’t true, then increasing the probability that the output is about an animal would also increase the relative probability that the output is about a dog rather than a cat. In practice, such exact representations are unlikely to be found by gradient descent in LLM training. Rather, we expect ℓ¯W⊤g(y)superscriptsubscript¯ℓtop _W g(y)over¯ start_ARG ℓ end_ARGW⊤ g ( y ) to be isotropically distributed around bwsubscriptb_wbitalic_w and 00, with variances that are small compared to bwsubscriptb_wbitalic_w (so that animal and non-animal words are well-separated.) With this result in hand, we can define a notion of vector representation for binary features: Definition 5. We say that a binary feature W for an attribute w has a vector representation ℓ¯w∈ℝdsubscript¯ℓsuperscriptℝ _w ^dover¯ start_ARG ℓ end_ARGw ∈ blackboard_Rd if ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw satisfies Definition 3 and ‖ℓ¯w‖2=bwsubscriptnormsubscript¯ℓ2subscript\| _w\|_2=b_w∥ over¯ start_ARG ℓ end_ARGw ∥2 = bitalic_w in Theorem 4. If the vector representation of a binary feature is not unique, we say ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw is the vector representation that maximizes bwsubscriptb_wbitalic_w. We have now moved from representations as directions to representations as vectors. Definition 5 and Theorem 4 give a simple way of composing binary features into binary contrasts: Corollary 6 (Binary Contrasts Are Vector Differences of Binary Features). Let w0⇒w1⇒subscript0subscript1w_0 w_1w0 ⇒ w1 be a binary contrast, and suppose there exist vector representations ℓ¯w0subscript¯ℓsubscript0 _w_0over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and ℓ¯w1subscript¯ℓsubscript1 _w_1over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT for each attribute w0subscript0w_0w0 and w1subscript1w_1w1. Then, the difference ℓ¯w1−ℓ¯w0subscript¯ℓsubscript1subscript¯ℓsubscript0 _w_1- _w_0over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT is a linear representation ℓ¯w0⇒w1subscript¯ℓ⇒subscript0subscript1 _w_0 w_1over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 ⇒ w1 end_POSTSUBSCRIPT in the sense of Definition 3. We can also apply ordinary vector space operations to construct representation of categorical concepts, e.g., mammal,reptile,bird,fish.mammalreptilebirdfish\ mammal, reptile, bird, fish\. mammal , reptile , bird , fish . There is now a straightforward way to define the representation of such concepts: Definition 7. The polytope representation of a categorical concept W=w0,…,wk−1subscript0…subscript1W=\w_0,…,w_k-1\W = w0 , … , witalic_k - 1 is the convex hull of the vector representations of the elements of the concept.333In Appendix C, we prove that the polytope representation for a “natural” categorical concept is a simplex. In Appendix D, we state and prove a generalization of Corollary 6 to categorical concepts (which proves Corollary 6 itself). Figure 2: Hierarchical semantics are encoded as orthogonality in the representation space (Theorem 8). The plots show the projection of the unembedding vectors onto 2D subspaces: spanℓ¯animal,ℓ¯mammalspansubscript¯ℓanimalsubscript¯ℓmammalspan\ _ animal, _ mammal\span over¯ start_ARG ℓ end_ARGanimal , over¯ start_ARG ℓ end_ARGmammal (left; (a)), spanℓ¯animal,ℓ¯bird−ℓ¯mammalspansubscript¯ℓanimalsubscript¯ℓbirdsubscript¯ℓmammalspan\ _ animal, _ bird- % _ mammal\span over¯ start_ARG ℓ end_ARGanimal , over¯ start_ARG ℓ end_ARGbird - over¯ start_ARG ℓ end_ARGmammal (middle; (b)), and spanℓ¯animal−ℓ¯plant,ℓ¯bird−ℓ¯mammalspansubscript¯ℓanimalsubscript¯ℓplantsubscript¯ℓbirdsubscript¯ℓmammalspan\ _ animal- _ plant, % _ bird- _ mammal\span over¯ start_ARG ℓ end_ARGanimal - over¯ start_ARG ℓ end_ARGplant , over¯ start_ARG ℓ end_ARGbird - over¯ start_ARG ℓ end_ARGmammal (right; (c)). Gray points indicate all 256K tokens in the vocabulary, and the colored points are the tokens in (w)Y(w)Y ( w ). The blue and red vectors are used to span the 2D subspaces. 4.2 Hierarchical Orthogonality Now, we turn to the question of how hierarchical relationships between concepts are encoded in the representation space. The core intuition is that manipulating the “animal” concept should not affect relative probabilities of the “mammal” and “bird” concepts, so we might expect the representations of animaltypewriter_animal and mammal⇒bird⇒mammalbird mammal birdmammal ⇒ bird to be orthogonal. The following result formalizes this intuition by connecting the vector and semantic structures. The result is illustrated in Figure 2. Theorem 8 (Hierarchical Orthogonality). Suppose there exist the vector representations for all the following binary features. Then, we have that (a) ℓ¯w⟂ℓ¯z−ℓ¯wperpendicular-tosubscript¯ℓsubscript¯ℓsubscript¯ℓ _w _z- _wover¯ start_ARG ℓ end_ARGw ⟂ over¯ start_ARG ℓ end_ARGz - over¯ start_ARG ℓ end_ARGw for z≺wprecedesz wz ≺ w; (b) ℓ¯w⟂ℓ¯z1−ℓ¯z0perpendicular-tosubscript¯ℓsubscript¯ℓsubscript1subscript¯ℓsubscript0 _w _z_1- _z_0over¯ start_ARG ℓ end_ARGw ⟂ over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT for Z∈Rz0,z1subscriptsubscript0subscript1Z _R\z_0,z_1\Z ∈R z0 , z1 subordinate to W∈Rnot_w,is_wsubscriptnot_wis_wW _R\ not\_w, is\_w\W ∈R not_w , is_w ; (c) ℓ¯w1−ℓ¯w0⟂ℓ¯z1−ℓ¯z0perpendicular-tosubscript¯ℓsubscript1subscript¯ℓsubscript0subscript¯ℓsubscript1subscript¯ℓsubscript0 _w_1- _w_0 _z_1- _z_0over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟂ over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT for Z∈Rz0,z1subscriptsubscript0subscript1Z _R\z_0,z_1\Z ∈R z0 , z1 subordinate to W∈Rw0,w1subscriptsubscript0subscript1W _R\w_0,w_1\W ∈R w0 , w1 ; and (d) ℓ¯w1−ℓ¯w0⟂ℓ¯w2−ℓ¯w1perpendicular-tosubscript¯ℓsubscript1subscript¯ℓsubscript0subscript¯ℓsubscript2subscript¯ℓsubscript1 _w_1- _w_0 _w_2- _w_1over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ⟂ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT for w2≺w1≺w0precedessubscript2subscript1precedessubscript0w_2 w_1 w_0w2 ≺ w1 ≺ w0. Together, Definition 7 and Theorem 8 give the simple structure illustrated in Figure 1: hierarchical concepts are represented as direct sums of polytopes. This direct sum structure is immediate from Theorem 8. We emphasize that all of the results—involving differences of representations—are only possible because we have vector representations (mere directions would not suffice). 5 Experiments We now turn to empirically testing the theoretical results. Here, we present our empirical results on the Gemma-2B model (Mesnard et al., 2024).444Code is available at github.com/KihoPark/LLM_Categorical_Hierarchical_Representations. In Appendix F, we further present empirical results on the LLaMA-3-8B model (Dubey et al., 2024), for which our findings are largely analogous. 5.1 Setup Canonical Representation The results in this paper rely on transforming the representation spaces so that the Euclidean inner product is a causal inner product, aligning the embedding and unembedding representations. Following Park et al. (2024), we estimate the required transformation as: g(y)=Cov(γ)−1/2(γ(y)−[γ])Covsuperscript12delimited-[]g(y)=Cov(γ)^-1/2(γ(y)-E[γ])g ( y ) = Cov ( γ )- 1 / 2 ( γ ( y ) - blackboard_E [ γ ] ) (5.1) where γ is the unembedding vector of a word sampled uniformly from the vocabulary. Centering by [γ]delimited-[]E[γ]blackboard_E [ γ ] is a reasonable approximation of centering by γ¯0subscript¯0 γ_0over¯ start_ARG γ end_ARG0 defined in Theorem 4 because this makes the projection of a random g(y)g(y)g ( y ) on an arbitrary direction close to 00. This matches the requirement that the projection of a word onto a concept the word does not belong to should be close to 00. WordNet We define a large collection of binary concepts using WordNet (Miller, 1995). WordNet organizes English words into a hierarchy of synsets, where each synset is a set of synonyms. The WordNet hierarchy is based on word hyponym relations, and reflects the semantic hierarchy of interest in this paper. We take each synset as an attribute w and define (w)Y(w)Y ( w ) as the collection of all words belonging to any synset that is a descendant of w. For example, the synset mammal.n.01 is a descendant of animal.n.01, so both (mammal.n.01)mammal.n.01Y( mammal.n.01)Y ( mammal.n.01 ) and (animal.n.01)animal.n.01Y( animal.n.01)Y ( animal.n.01 ) contain the word “dog”. We collect all noun and verb synsets, and augment the word collections by including plural forms of the nouns, multiple tenses of each verb, and capital and lower case versions of each word. We filter to include only those synsets with at least 50 words in the Gemma vocabulary. This leaves us with 593 noun and 364 verb synsets, each defining an attribute. For space, we report results on the noun hierarchy here and defer the verb hierarchy to Appendix F. Estimation via Linear Discriminant Analysis Now, we want to estimate the vector representation ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw for each attribute w. To do this, we make use of vocabulary sets (w)Y(w)Y ( w ). Following Theorem 4, the vector associated to the concept w should have two properties. First, when the full vocabulary is projected onto this vector, the words in (w)Y(w)Y ( w ) should be well-separated from the rest of the vocabulary. Second, the projection of the unembedding vectors for y∈(w)y (w)y ∈ Y ( w ) should be approximately the same value. Equivalently, the variance of the projection of the unembedding vectors for y∈(w)y (w)y ∈ Y ( w ) should be small. To capture these requirements, we estimate the directions using a variant of Linear Discriminant Analysis (LDA), which finds a projection minimizing within-class variance and maximizing between-class variance. Formally, we estimate the vector representation of a binary feature W for an attribute w as ℓ¯w=(g~w⊤(gw))g~w,withg~w=Cov(gw)†(gw)‖Cov(gw)†(gw)‖2,formulae-sequencesubscript¯ℓsuperscriptsubscript~topsubscriptsubscript~withsubscript~Covsuperscriptsubscript†subscriptsubscriptnormCovsuperscriptsubscript†subscript2 _w= ( g_w E(g_w) ) g_w% , g_w= Cov(g_w) % E(g_w)\|Cov(g_w) E(g_w)\|_2,over¯ start_ARG ℓ end_ARGw = ( over~ start_ARG g end_ARGw⊤ blackboard_E ( gitalic_w ) ) over~ start_ARG g end_ARGw , with over~ start_ARG g end_ARGw = divide start_ARG Cov ( gitalic_w )† blackboard_E ( gitalic_w ) end_ARG start_ARG ∥ Cov ( gitalic_w )† blackboard_E ( gitalic_w ) ∥2 end_ARG , (5.2) where gwsubscriptg_wgitalic_w is the unembedding vector of a word sampled uniformly from (w)Y(w)Y ( w ) and Cov(gw)†Covsuperscriptsubscript†Cov(g_w) Cov ( gitalic_w )† is a pseudo-inverse of the covariance matrix. We estimate the covariance matrix Cov(gw)CovsubscriptCov(g_w)Cov ( gitalic_w ) using the Ledoit-Wolf shrinkage estimator (Ledoit & Wolf, 2004), because the dimension of the representation spaces is much higher than the number of samples. 5.2 WordNet Hierarchy is Linearly Represented Figure 3: Vector representations exist for most binary features in the WordNet noun hierarchy. For each synset w (indexed on the x-axis) we estimate the vector representation ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw using a train subset of the vocabulary (w)Y(w)Y ( w ). The plot shows the projections (g(y)⊤ℓ¯w)/‖ℓ¯w‖22superscripttopsubscript¯ℓsuperscriptsubscriptnormsubscript¯ℓ22(g(y) _w)/\| _w\|_2^2( g ( y )⊤ over¯ start_ARG ℓ end_ARGw ) / ∥ over¯ start_ARG ℓ end_ARGw ∥22 of train (green), test (blue), and random (orange) words on estimated vector representations for each WordNet feature, using either the original (left) or shuffled (right) unembeddings. Our theory predicts that this value should be close to 1 when y has the target feature, and close to 0 when it does not. The thick lines present the mean of the projections for each feature and the error bars indicate the standard deviation. As predicted, the projections of test words are near 1, and random words near 0 (left plot). Further, this structure does not hold when using the shuffled control without natural semantics (right plot). Existence of Vector Representations for Binary Features The first question is whether vector representations of binary features exist. To evaluate this, for each synset w in WordNet we split (w)Y(w)Y ( w ) into train words (70%) and test words (30%), fit the LDA estimator to the train words, and examine the projection of the unembedding vectors for the test and random words onto the estimated vector representation. The left plot in Figure 3 shows the mean and standard deviation of the projections, divided by the magnitude of each estimated ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw. Following Theorem 4, if a vector representation exists for an concept, we would expect the values on the test set to be close to 1, and the values for random words to be close to 0. We see that this is indeed the case, giving evidence that vector representations do exist for these features. As a baseline, the right plot in Figure 3 shows the same analysis but with the unembedding vectors randomly shuffled. In this case, the test projections are close to 0, which indicates that there are no linear representations. Thus, the existence of the linear representations in the original unembeddings relies on the underlying semantic structure. Table 1: Adding the linear representation of the parent concept to context embeddings does not affect the logit differences between token pairs in a child concept, and it substantially affects those in the parent concept. For each parent-child pair of binary concepts (w0⇒w1,z0⇒z1)formulae-sequence⇒subscript0subscript1⇒subscript0subscript1(w_0 w_1,z_0 z_1)( w0 ⇒ w1 , z0 ⇒ z1 ), we first add the normalized linear representation ℓ¯W=ℓ¯w1−ℓ¯w0subscript¯ℓsubscript¯ℓsubscript1subscript¯ℓsubscript0 _W= _w_1- _w_0over¯ start_ARG ℓ end_ARGW = over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to the context embeddings. Then, we show the change in logit difference logℙ(y1|ℓ)ℙ(y0|ℓ)ℙconditionalsubscript1ℓℙconditionalsubscript0ℓ P(y_1~|~ )P(y_0~|~ )log divide start_ARG blackboard_P ( y1 | ℓ ) end_ARG start_ARG blackboard_P ( y0 | ℓ ) end_ARG between the pairs (y0,y1)∈(w0)×(w1)subscript0subscript1subscript0subscript1(y_0,y_1) (w_0)×Y(w_1)( y0 , y1 ) ∈ Y ( w0 ) × Y ( w1 ) (parent; top row) and (y0,y1)∈(z0)×(z1)subscript0subscript1subscript0subscript1(y_0,y_1) (z_0)×Y(z_1)( y0 , y1 ) ∈ Y ( z0 ) × Y ( z1 ) (child; bottom row). Notice that, for any context x and any tokens y0subscript0y_0y0 and y1subscript1y_1y1, adding a linear representation ℓ¯Wsubscript¯ℓ _Wover¯ start_ARG ℓ end_ARGW for a binary contrast W=w0⇒w1subscript0⇒subscript1W=w_0 w_1W = w0 ⇒ w1 manipulates the logit difference between the tokens from ℓ(x)⊤(g(y1)−g(y0))ℓsuperscripttopsubscript1subscript0 (x) (g(y_1)-g(y_0))ℓ ( x )⊤ ( g ( y1 ) - g ( y0 ) ) to (ℓ(x)+ℓ¯W)⊤(g(y1)−g(y0))superscriptℓsubscript¯ℓtopsubscript1subscript0( (x)+ _W) (g(y_1)-g(y_0))( ℓ ( x ) + over¯ start_ARG ℓ end_ARGW )⊤ ( g ( y1 ) - g ( y0 ) ), which implies the change in logit difference between the tokens is ℓ¯W⊤(g(y1)−g(y0))superscriptsubscript¯ℓtopsubscript1subscript0 _W (g(y_1)-g(y_0))over¯ start_ARG ℓ end_ARGW⊤ ( g ( y1 ) - g ( y0 ) ), irrespective of ℓ(x)ℓ (x)ℓ ( x ). We show the mean and standard deviation of the change in logit differences over all token pairs. W=parent0⇒parent1subscriptparent0⇒subscriptparent1W=parent_0 _1W = parent0 ⇒ parent1 & Z=child0⇒child1subscriptchild0⇒subscriptchild1Z=child_0 _1Z = child0 ⇒ child1 Change in Logit Differences W=plant.n.02⇒animal.n.01plant.n.02⇒animal.n.01W= plant.n.02 animal.n.01W = plant.n.02 ⇒ animal.n.01 5.1265±1.1731plus-or-minus5.12651.17315.1265± 1.17315.1265 ± 1.1731 Z=mammal.n.01⇒reptile.n.01mammal.n.01⇒reptile.n.01Z= mammal.n.01 reptile.n.01Z = mammal.n.01 ⇒ reptile.n.01 −0.0600±1.2190plus-or-minus0.06001.2190-0.0600± 1.2190- 0.0600 ± 1.2190 W=fluid.n.02⇒solid.n.01fluid.n.02⇒solid.n.01W= fluid.n.02 solid.n.01W = fluid.n.02 ⇒ solid.n.01 9.8296±1.1099plus-or-minus9.82961.10999.8296± 1.10999.8296 ± 1.1099 Z=crystal.n.01⇒food.n.02crystal.n.01⇒food.n.02Z= crystal.n.01 food.n.02Z = crystal.n.01 ⇒ food.n.02 0.3770±1.5410plus-or-minus0.37701.54100.3770± 1.54100.3770 ± 1.5410 W=scientist.n.01⇒contestant.n.01scientist.n.01⇒contestant.n.01W= scientist.n.01 contestant.n.01W = scientist.n.01 ⇒ contestant.n.01 14.4222±0.9458plus-or-minus14.42220.945814.4222± 0.945814.4222 ± 0.9458 Z=athlete.n.01⇒player.n.01athlete.n.01⇒player.n.01Z= athlete.n.01 player.n.01Z = athlete.n.01 ⇒ player.n.01 −0.1545±1.1426plus-or-minus0.15451.1426-0.1545± 1.1426- 0.1545 ± 1.1426 Intervention Next, we validate that adding the estimated linear representations of a binary contrast changes the target concept without changing other off-target concepts, as required by Definition 3. Table 1 shows the mean and standard deviation of the changes in the logit differences between the pairs from parent or child binary contrasts, after adding the normalized linear representations of the parent binary contrast to context embeddings. The results show that the logit differences change significantly for the target concept, while the off-target concept changes very little. Figure 4: Hierarchical semantics in WordNet are linearly represented in Gemma-2B. The left heatmap shows pairwise shortest distance matrix between features in the noun hierarchy graph as (1+min_distance)−1superscript1min_distance1(1+min\_distance)^-1( 1 + min_distance )- 1 (higher values indicate closeness, such as in child-parent or sibling relationships). The middle heatmap shows the cosine similarity between the vector representations ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw. As predicted, this similarity reflects the WordNet structure. The right heatmap is a control where the embeddings are randomly shuffled (removing semantic structure). In this case, nearly everything is orthogonal, as expected in high-dimensional space (set inclusion relationships remain due to the estimation procedure). In Appendix F, we include zoomed-in versions of these heatmaps. Relationship Between the Vector Representations of Binary Features We now turn to examining whether the similarity between the vector representations of binary features reflects their semantic relation. The direct sum structure predicts that concepts that are close in semantic hierarchy should have similar vector representations. For example, mammaltypewriter_mammal and birdtypewriter_bird are close in the hyponym graph because they share a common parent ( animaltypewriter_animal). Our theory predicts that ℓ¯birdsubscript¯ℓbird _ birdover¯ start_ARG ℓ end_ARGbird and ℓ¯mammalsubscript¯ℓmammal _ mammalover¯ start_ARG ℓ end_ARGmammal share a common component ℓ¯animalsubscript¯ℓanimal _ animalover¯ start_ARG ℓ end_ARGanimal (the representation for “bird” is the representation for “animal” plus an orthogonal component, and similarly for mammal). This implies that the cosine similarity between ℓ¯mammalsubscript¯ℓmammal _ mammalover¯ start_ARG ℓ end_ARGmammal and ℓ¯birdsubscript¯ℓbird _ birdover¯ start_ARG ℓ end_ARGbird should be substantial. In Figure 4, we show the shortest distance between each feature in the (undirected) WordNet noun hyponym graph in the left panel and the cosine similarity between the estimated vector representations ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw in the middle panel. It is clear that, as predicted, the cosine similarity reflects the WordNet structure. As a control, we apply the same analysis after randomly shuffling the embeddings and show the resulting cosine similarities in the right panel. Here, direct parent-child (or grandparent-grandchild) relationships are still reflected in the cosine similarity because the shuffled embeddings still respect set inclusion (the words assigned to “mammal” are still a subset of those assigned to “animal”). However, sibling relationships are not reflected once the semantic structure is removed by the shuffling. In this case, the representations of siblings are effectively pairs of random vectors, and are nearly orthogonal as we would expect in a high dimensional space. Figure 5: WordNet noun hierarchy is encoded in the orthogonal structure predicted by statement (a) in Theorem 8. We plot the cosine similarity between a child-parent vector and a parent vector for each feature in the hierarchy (blue). As predicted, this value is close to 0. The left plot uses all data for representation estimation, and the right plot uses only 70% independently selected for each synset. We include baselines where a randomly selected feature is used as the parent (orange) and where the embeddings are shuffled (green) as controls for the possibility that the orthogonality is a simple byproduct of high-dimensional geometry, or of the set inclusion relationships used in estimation—see main text for details. See Appendix F for an analogous plot for statement (d). Hierarchical Orthogonality Finally, we evaluate the prediction that hierarchical relations are encoded orthogonally as predicted in Theorem 8. Figure 5 shows the cosine similarity ℓ¯parentsubscript¯ℓparent _ parentover¯ start_ARG ℓ end_ARGparent and ℓ¯child−ℓ¯parentsubscript¯ℓchildsubscript¯ℓparent _ child- _ parentover¯ start_ARG ℓ end_ARGchild - over¯ start_ARG ℓ end_ARGparent for the WordNet features. As predicted, this value is close to 0 (blue curve). Now, a challenge here is that in high dimensional spaces even random vectors are nearly orthogonal. So, it may be difficult to differentiate whether orthogonality reflects semantic structure or is merely a consequence of nearly everything being orthogonal in high dimensions. As a control, we also show the cosine similarity when the parent vector is the representation of a randomly selected feature (orange). In this case, the cosine similarity is far from 0, suggesting that the observed orthogonality is not merely a byproduct of the high-dimensional geometry. We also include another baseline that estimates the cosine similarity between the child-parent vector and the parent vector for each feature using shuffled unembeddings (green). In the left plot, we see that we still have orthogonality, which could suggest that the orthogonality is a consequence of the set inclusion relations in WordNet (rather than actual semantic structure).555In Appendix G, we explain why set inclusion (before train/test split) leads to orthogonality. To test this, in the right plot we estimate the representations using only 70% of the tokens, independently selected for each synset. This breaks the set inclusion. In this case, we see the orthogonality is preserved for the original unembeddings, but is broken for the shuffled unembeddings. Note that a cosine of −0.20.2-0.2- 0.2 is highly nontrivial in a high-dimensional space. This suggests the orthogonality does indeed reflect the semantic structure. 6 Discussion and Related Work We set out to understand how semantic structure is encoded in the geometry of representation space. We have arrived at an astonishingly simple structure, summarized in Figure 1. The key contributions are moving from representing concepts as directions to representing them as vectors (and polytopes), and connecting semantic hierarchy to orthogonality. Related Work The results here connect closely to the study of linear representations in language models (e.g., Mikolov et al., 2013; Pennington et al., 2014; Arora et al., 2016; Elhage et al., 2022; Burns et al., 2022; Tigges et al., 2023; Nanda et al., 2023; Moschella et al., 2022; Li et al., 2023; Gurnee et al., 2023; Wang et al., 2023; Jiang et al., 2024; Park et al., 2024). In particular, Park et al. (2024) formalize the linear representation hypothesis by unifying three distinct notions of linearity: word2vec-like embedding differences, logistic probing, and steering vectors. Our work relies on this unification, and just focuses on the steering vector notion. Our work also connects to work aimed at theoretically understanding the existence of linear representations. These include early work on word2vec-style embedding models (Arora et al., 2016; Gittens et al., 2017; Arora et al., 2018; Ethayarajh et al., 2018; Frandsen & Ge, 2019; Allen & Hospedales, 2019) as well as dynamic topic models (Blei & Lafferty, 2006; Rudolph et al., 2016). Jiang et al. (2024) connect the existence of linear representations in LLMs to the implicit bias of gradient descent. In this paper, we do not seek to justify the existence of linear representations, but rather to understand their structure if they do exist. Though, by empirically estimating vector representations for thousands of concepts, we add to the body of evidence supporting the existence of linear representations. Elhage et al. (2022) also empirically observe the formation of polytopes in the representation space of a toy model, and the present work can be viewed in part as giving an explanation for this phenomenon. There is also a growing literature studying the representation geometry of natural language (Mimno & Thompson, 2017; Reif et al., 2019; Volpi & Malagò, 2021; Li et al., 2020; Chen et al., 2021; Chang et al., 2022; Liang et al., 2022; Jiang et al., 2023; Wang et al., 2023; Park et al., 2024; Valeriani et al., 2024). In terms of hierarchical structures, existing work focuses on connections to hyperbolic geometry (Nickel & Kiela, 2017; Ganea et al., 2018; Chen et al., 2021; He et al., 2024). We do not find such a connection in LLMs, but it is an interesting direction for future work to determine if more efficient LLM representations could be constructed in hyperbolic space. Jiang et al. (2023) hypothesize that very general ”independence structures” are naturally represented by partial orthogonality in vector spaces (Amini et al., 2022). The results here confirm and expand on this hypothesis in the case of hierarchical structure in language models. Implications and Future Work The results in this paper give a foundational understanding the structure of representation space in language models. Of course, the ultimate purpose of foundations is to build upon them. One immediate direction is to refine the attempts to interpret LLM structure to explicitly account for hierarchical semantics. As an example, there is currently significant interest in using sparse autoencoders to extract interpretable features from LLMs (e.g., Cunningham et al., 2023; Bricken et al., 2023; Kissane et al., 2024; Braun et al., 2024). This work searches for representations in terms of distinct binary features. Concretely, it hopes to find features for, e.g., animaltypewriter_animal, mammaltypewriter_mammal, birdtypewriter_bird, etc. Based on the results here, these representations are strongly co-linear, and potentially difficult to disentangle. On the other hand, a representation in terms of ℓ¯animalsubscript¯ℓanimal _ animalover¯ start_ARG ℓ end_ARGanimal, ℓ¯mammal−ℓ¯animalsubscript¯ℓmammalsubscript¯ℓanimal _ mammal- _ animalover¯ start_ARG ℓ end_ARGmammal - over¯ start_ARG ℓ end_ARGanimal, ℓ¯bird−ℓ¯animalsubscript¯ℓbirdsubscript¯ℓanimal _ bird- _ animalover¯ start_ARG ℓ end_ARGbird - over¯ start_ARG ℓ end_ARGanimal, etc., will be cleanly separated and equally interpretable. Fundamentally, semantic meaning has hierarchical structure, so interpretability methods should respect this structure. Understanding the geometric representation makes it possible to design such methods. In a separate, foundational, direction: the results in this paper rely on using the canonical representation space, and we estimate this using the whitening transformation of the unembedding layer. However, this technique only works for the final layer representation. It is an important open question how to make sense of the geometry of internal layers. Acknowledgments This work is supported by ONR grant N00014-23-1-2591 and Open Philanthropy. References Allen & Hospedales (2019) Carl Allen and Timothy Hospedales. Analogies explained: Towards understanding word embeddings. In International Conference on Machine Learning, p. 223–231. PMLR, 2019. Amini et al. (2022) Arash A Amini, Bryon Aragam, and Qing Zhou. A non-graphical representation of conditional independence via the neighbourhood lattice. arXiv preprint arXiv:2206.05829, 2022. Arora et al. (2016) Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics, 4:385–399, 2016. Arora et al. (2018) Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483–495, 2018. Blei & Lafferty (2006) David M Blei and John D Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, p. 113–120, 2006. Braun et al. (2024) Dan Braun, Jordan Taylor, Nicholas Goldowsky-Dill, and Lee Sharkey. Identifying functionally important features with end-to-end sparse dictionary learning. arXiv preprint arXiv:2405.12241, 2024. Bricken et al. (2023) Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html. Burns et al. (2022) Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. arXiv preprint arXiv:2212.03827, 2022. Chang et al. (2022) Tyler A Chang, Zhuowen Tu, and Benjamin K Bergen. The geometry of multilingual language model representations. arXiv preprint arXiv:2205.10964, 2022. Chen et al. (2021) Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. Probing bert in hyperbolic spaces. arXiv preprint arXiv:2104.03869, 2021. Cunningham et al. (2023) Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Elhage et al. (2022) Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022. Ethayarajh et al. (2018) Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. Towards understanding linear word analogies. arXiv preprint arXiv:1810.04882, 2018. Frandsen & Ge (2019) Abraham Frandsen and Rong Ge. Understanding composition of word embeddings via tensor decomposition. arXiv preprint arXiv:1902.00613, 2019. Ganea et al. (2018) Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In International Conference on Machine Learning, p. 1646–1655. PMLR, 2018. Gittens et al. (2017) Alex Gittens, Dimitris Achlioptas, and Michael W. Mahoney. Skip-gram - zipf + uniform = vector additivity. In Annual Meeting of the Association for Computational Linguistics, 2017. Gurnee & Tegmark (2024) Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=jE8xbmvFin. Gurnee et al. (2023) Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023. He et al. (2024) Yuan He, Zhangdie Yuan, Jiaoyan Chen, and Ian Horrocks. Language models as hierarchy encoders. arXiv preprint arXiv:2401.11374, 2024. Jiang et al. (2023) Yibo Jiang, Bryon Aragam, and Victor Veitch. Uncovering meanings of embeddings via partial orthogonality. arXiv preprint arXiv:2310.17611, 2023. Jiang et al. (2024) Yibo Jiang, Goutham Rajendran, Pradeep Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. In International Conference on Machine Learning, 2024. Kissane et al. (2024) Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. Sparse autoencoders work on attention layer outputs. Alignment Forum, 2024. URL https://w.alignmentforum.org/posts/DtdzGwFh9dCfsekZZ. Ledoit & Wolf (2004) Olivier Ledoit and Michael Wolf. A well-conditioned estimator for large-dimensional covariance matrices. Journal of multivariate analysis, 88(2):365–411, 2004. Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. arXiv preprint arXiv:2011.05864, 2020. Li et al. (2023) Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. arXiv preprint arXiv:2306.03341, 2023. Liang et al. (2022) Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612–17625, 2022. Mesnard et al. (2024) Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. Mikolov et al. (2013) Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, p. 746–751, 2013. Miller (1995) George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41, 1995. Mimno & Thompson (2017) David Mimno and Laure Thompson. The strange geometry of skip-gram with negative sampling. In Conference on Empirical Methods in Natural Language Processing, 2017. Moschella et al. (2022) Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodola. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430, 2022. Nanda et al. (2023) Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023. Nickel & Kiela (2017) Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems, 30, 2017. OpenAI (2023) OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Park et al. (2024) Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning, 2024. Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), p. 1532–1543, 2014. Reif et al. (2019) Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. Advances in Neural Information Processing Systems, 32, 2019. Rudolph et al. (2016) Maja Rudolph, Francisco Ruiz, Stephan Mandt, and David Blei. Exponential family embeddings. Advances in Neural Information Processing Systems, 29, 2016. Tigges et al. (2023) Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models. arXiv preprint arXiv:2310.15154, 2023. Valeriani et al. (2024) Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models. Advances in Neural Information Processing Systems, 36, 2024. Volpi & Malagò (2021) Riccardo Volpi and Luigi Malagò. Natural alpha embeddings. Information Geometry, 4(1):3–29, 2021. Wang et al. (2023) Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text-controlled generative models. In Advances in Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=SGlrCuwdsB. Figure 6: Categorical concepts are represented as polytopes. The plots show the projection of the unembedding vectors on the 3D subspaces: spanℓ¯mammal,ℓ¯bird,ℓ¯fishspansubscript¯ℓmammalsubscript¯ℓbirdsubscript¯ℓfishspan\ _ mammal, _ bird, % _ fish\span over¯ start_ARG ℓ end_ARGmammal , over¯ start_ARG ℓ end_ARGbird , over¯ start_ARG ℓ end_ARGfish (left) and spanℓ¯bird−ℓ¯mammal,ℓ¯fish−ℓ¯mammal,ℓ¯reptile−ℓ¯mammalspansubscript¯ℓbirdsubscript¯ℓmammalsubscript¯ℓfishsubscript¯ℓmammalsubscript¯ℓreptilesubscript¯ℓmammalspan\ _ bird- _ mammal, % _ fish- _ mammal, _ reptile% - _ mammal\span over¯ start_ARG ℓ end_ARGbird - over¯ start_ARG ℓ end_ARGmammal , over¯ start_ARG ℓ end_ARGfish - over¯ start_ARG ℓ end_ARGmammal , over¯ start_ARG ℓ end_ARGreptile - over¯ start_ARG ℓ end_ARGmammal (right). The gray points indicate all 256K tokens in the vocabulary, and the colored points are the tokens in (w)wY( w)Y ( w ). The left plot further shows the orthogonality between the triangle and the projection of ℓ¯animalsubscript¯ℓanimal _ animalover¯ start_ARG ℓ end_ARGanimal (black arrow). Appendix A Visualization of animaltypewriter_animal As a concrete example for visualizations, we examine the theoretical predictions for the familiar concept animaltypewriter_animal. However, the categories in WordNet are not the most intuitive; for example, the subcategories under textttanimal.n.01 include terms like “chordate”, “aquatic vertebrate”, “invertebrate” instead of more familiar categories such as “fish”, “amphibian”, or “insect”. This makes it challenging to gather tokens corresponding to clear, high-level concepts mammal,bird,fish,reptile,amphibian,insectmammalbirdfishreptileamphibianinsect\ mammal, bird, fish, reptile, % amphibian, insect\ mammal , bird , fish , reptile , amphibian , insect . Therefore, we generated two sets of tokens (animal)animalY( animal)Y ( animal ) and (plant)plantY( plant)Y ( plant ) using ChatGPT-4 (OpenAI, 2023), and manually inspected them. (animal)animalY( animal)Y ( animal ) is further divided into six sets of tokens for each subcategory mammal,bird,fish,reptile,amphibian,insectmammalbirdfishreptileamphibianinsect\ mammal, bird, fish, reptile, % amphibian, insect\ mammal , bird , fish , reptile , amphibian , insect . Figure 2 illustrates the geometric relationships between various representation vectors. The main takeaway is that the semantic hierarchy is encoded as orthogonality in the manner predicted by Theorem 8. The figure also illustrates Theorem 4, showing that the projection of the unembedding vectors for y∈(w)y (w)y ∈ Y ( w ) is approximately constant, while the projection of y∉(w)y (w)y ∉ Y ( w ) is zero. Figure 6 illustrates that the representation of a categorical concept is a polytope. and the projection of unembedding vectors onto the subspace for the polytope are concentrated on each vertex as predicted in Corollary 10. The left plot also shows that, as predicted, the polytope for fish,\ fish, fish , mammal,mammal mammal,mammal , bird bird\bird is orthogonal to the vector representation of animaltypewriter_animal. Appendix B Proofs B.1 Proof of Theorem 4 See 4 Proof. For any y1,y0∈(w)subscript1subscript0y_1,y_0 (w)y1 , y0 ∈ Y ( w ) or y1,y0∉(w)subscript1subscript0y_1,y_0 (w)y1 , y0 ∉ Y ( w ), let Z be a binary concept where (Z=0)=y00subscript0Y(Z=0)=\y_0\Y ( Z = 0 ) = y0 and (Z=1)=y11subscript1Y(Z=1)=\y_1\Y ( Z = 1 ) = y1 . Since Z is subordinate to W, eq. 3.2 implies that logitℙ(Y=y1|Y∈y0,y1,ℓ+ℓ¯W)=logitℙ(Y=y1|Y∈y0,y1,ℓ) *logitP(Y=y_1~|~Y∈\y_0,y_1\% , + _W)= *logitP(Y=y_1~|~Y∈\y_0% ,y_1\, )logit blackboard_P ( Y = y1 | Y ∈ y0 , y1 , ℓ + over¯ start_ARG ℓ end_ARGW ) = logit blackboard_P ( Y = y1 | Y ∈ y0 , y1 , ℓ ) (B.1) ⇔ℓ¯W⊤(g(y1)−g(y0))=ℓ¯W⊤A(γ(y1)−γ(y0))=0iffabsentsuperscriptsubscript¯ℓtopsubscript1subscript0superscriptsubscript¯ℓtopsubscript1subscript00 _W (g(y_1)-g(y_0))= _W % A(γ(y_1)-γ(y_0))=0⇔ over¯ start_ARG ℓ end_ARGW⊤ ( g ( y1 ) - g ( y0 ) ) = over¯ start_ARG ℓ end_ARGW⊤ A ( γ ( y1 ) - γ ( y0 ) ) = 0 (B.2) where A is the invertible matrix in eq. 2.2. This means that ℓ¯W⊤Aγ(y)superscriptsubscript¯ℓtop _W Aγ(y)over¯ start_ARG ℓ end_ARGW⊤ A γ ( y ) is the same for all y∈(w)y (w)y ∈ Y ( w ), and it is also the same for all y∉(w)y (w)y ∉ Y ( w ). Furthermore, for any y1∈(w)subscript1y_1 (w)y1 ∈ Y ( w ) and y0∉(w)subscript0y_0 (w)y0 ∉ Y ( w ), eq. 3.1 implies that logitℙ(Y=y1|Y∈y0,y1,ℓ+ℓ¯W)>logitℙ(Y=y1|Y∈y0,y1,ℓ) *logitP(Y=y_1~|~Y∈\y_0,y_1\% , + _W)> *logitP(Y=y_1~|~Y∈\y_0% ,y_1\, )logit blackboard_P ( Y = y1 | Y ∈ y0 , y1 , ℓ + over¯ start_ARG ℓ end_ARGW ) > logit blackboard_P ( Y = y1 | Y ∈ y0 , y1 , ℓ ) (B.3) ⇔ℓ¯W⊤(g(y1)−g(y0))=ℓ¯W⊤A(γ(y1)−γ(y0))>0.iffabsentsuperscriptsubscript¯ℓtopsubscript1subscript0superscriptsubscript¯ℓtopsubscript1subscript00 _W (g(y_1)-g(y_0))= _W % A(γ(y_1)-γ(y_0))>0.⇔ over¯ start_ARG ℓ end_ARGW⊤ ( g ( y1 ) - g ( y0 ) ) = over¯ start_ARG ℓ end_ARGW⊤ A ( γ ( y1 ) - γ ( y0 ) ) > 0 . (B.4) Thus, by setting bw0=ℓ¯W⊤Aγ(y)superscriptsubscript0superscriptsubscript¯ℓtopb_w^0= _W Aγ(y)bitalic_w0 = over¯ start_ARG ℓ end_ARGW⊤ A γ ( y ) for any y∉(w)y (w)y ∉ Y ( w ), and bw=ℓ¯W⊤Aγ(y1)−ℓ¯W⊤Aγ(y0)>0subscriptsuperscriptsubscript¯ℓtopsubscript1superscriptsubscript¯ℓtopsubscript00b_w= _W Aγ(y_1)- _W Aγ(y_0)>0bitalic_w = over¯ start_ARG ℓ end_ARGW⊤ A γ ( y1 ) - over¯ start_ARG ℓ end_ARGW⊤ A γ ( y0 ) > 0 for any y1∈(w)subscript1y_1 (w)y1 ∈ Y ( w ) and y0∉(w)subscript0y_0 (w)y0 ∉ Y ( w ), we get ℓ¯W⊤Aγ(y)=bw0+bwif y∈(w)ℓ¯W⊤Aγ(y)=bw0if y∉(w).casessuperscriptsubscript¯ℓtopsuperscriptsubscript0subscriptif superscriptsubscript¯ℓtopsuperscriptsubscript0if cases _W Aγ(y)=b_w^0+b_w&if y∈% Y(w)\\ _W Aγ(y)=b_w^0&if y (w).% cases start_ROW start_CELL over¯ start_ARG ℓ end_ARGW⊤ A γ ( y ) = bitalic_w0 + bitalic_w end_CELL start_CELL if y ∈ Y ( w ) end_CELL end_ROW start_ROW start_CELL over¯ start_ARG ℓ end_ARGW⊤ A γ ( y ) = bitalic_w0 end_CELL start_CELL if y ∉ Y ( w ) . end_CELL end_ROW (B.5) Then, we can choose an origin as γ¯0w=bw0A−1ℓ¯Wsuperscriptsubscript¯0superscriptsubscript0superscript1subscript¯ℓ γ_0^w=b_w^0A^-1 _Wover¯ start_ARG γ end_ARG0w = bitalic_w0 A- 1 over¯ start_ARG ℓ end_ARGW (B.6) satisfying eq. 4.1. On the other hand, if there exist d attributes w0,…,wd−1subscript0…subscript1\w_0,…,w_d-1\ w0 , … , witalic_d - 1 such that the linear representations ℓ¯W0,…,ℓ¯Wd−1subscript¯ℓsubscript0…subscript¯ℓsubscript1 _W_0,…, _W_d-1over¯ start_ARG ℓ end_ARGW start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , … , over¯ start_ARG ℓ end_ARGW start_POSTSUBSCRIPT d - 1 end_POSTSUBSCRIPT of the binary features W0,…,Wd−1subscript0…subscript1W_0,…,W_d-1W0 , … , Witalic_d - 1 for these attributes are linearly independent, then the linear system ℓ¯Wi⊤Aγ¯0=bwi0fori=0,…,d−1formulae-sequencesuperscriptsubscript¯ℓsubscripttopsubscript¯0superscriptsubscriptsubscript0for0…1 _W_i A γ_0=b_w_i^0 % for i=0,…,d-1over¯ start_ARG ℓ end_ARGW start_POSTSUBSCRIPT i end_POSTSUBSCRIPT⊤ A over¯ start_ARG γ end_ARG0 = bitalic_w start_POSTSUBSCRIPT i end_POSTSUBSCRIPT0 for i = 0 , … , d - 1 (B.7) has a unique solution γ¯0subscript¯0 γ_0over¯ start_ARG γ end_ARG0. We can choose this vector as the canonical origin γ¯0subscript¯0 γ_0over¯ start_ARG γ end_ARG0 in eq. 2.2, ensuring that eq. 4.1 is satisfied. ∎ B.2 Proof of Theorem 8 See 8 Proof. (a) For ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw and ℓ¯zsubscript¯ℓ _zover¯ start_ARG ℓ end_ARGz where z≺wprecedesz wz ≺ w, by Theorem 4, we have (ℓ¯z−ℓ¯w)⊤g(y)=bz−bwif y∈(z)(ℓ¯z−ℓ¯w)⊤g(y)=0−bw=−bwif y∈(w)∖(z)(ℓ¯z−ℓ¯w)⊤g(y)=0−0=0if y∉(w).casessuperscriptsubscript¯ℓsubscript¯ℓtopsubscriptsubscriptif superscriptsubscript¯ℓsubscript¯ℓtop0subscriptsubscriptif superscriptsubscript¯ℓsubscript¯ℓtop000if cases( _z- _w) g(y)=b_z-b_w&if % y (z)\\ ( _z- _w) g(y)=0-b_w=-b_w&if y∈% Y(w) (z)\\ ( _z- _w) g(y)=0-0=0&if y % (w). cases start_ROW start_CELL ( over¯ start_ARG ℓ end_ARGz - over¯ start_ARG ℓ end_ARGw )⊤ g ( y ) = bitalic_z - bitalic_w end_CELL start_CELL if y ∈ Y ( z ) end_CELL end_ROW start_ROW start_CELL ( over¯ start_ARG ℓ end_ARGz - over¯ start_ARG ℓ end_ARGw )⊤ g ( y ) = 0 - bitalic_w = - bitalic_w end_CELL start_CELL if y ∈ Y ( w ) ∖ Y ( z ) end_CELL end_ROW start_ROW start_CELL ( over¯ start_ARG ℓ end_ARGz - over¯ start_ARG ℓ end_ARGw )⊤ g ( y ) = 0 - 0 = 0 end_CELL start_CELL if y ∉ Y ( w ) . end_CELL end_ROW (B.8) When w∖zw zw ∖ z denotes an attribute defined by (w)∖(z)Y(w) (z)Y ( w ) ∖ Y ( z ), ℓ¯z−ℓ¯wsubscript¯ℓsubscript¯ℓ _z- _wover¯ start_ARG ℓ end_ARGz - over¯ start_ARG ℓ end_ARGw can change the target concept w∖z⇒z⇒w z zw ∖ z ⇒ z without changing any other concept subordinate or causally separable to the target concept. Thus, ℓ¯z−ℓ¯wsubscript¯ℓsubscript¯ℓ _z- _wover¯ start_ARG ℓ end_ARGz - over¯ start_ARG ℓ end_ARGw is the linear representation ℓ¯w∖z⇒zsubscript¯ℓ⇒ _w z zover¯ start_ARG ℓ end_ARGw ∖ z ⇒ z. This concept means not_z⇒is_z⇒not_zis_z not\_z is\_znot_z ⇒ is_z conditioned on w, and hence it is subordinate to w. Therefore, ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw is orthogonal to the linear representation ℓ¯w∖z⇒z=ℓ¯z−ℓ¯wsubscript¯ℓ⇒subscript¯ℓsubscript¯ℓ _w z z= _z- _wover¯ start_ARG ℓ end_ARGw ∖ z ⇒ z = over¯ start_ARG ℓ end_ARGz - over¯ start_ARG ℓ end_ARGw by the property of the causal inner product. If they are not orthogonal, adding ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw can change the other concept w∖z⇒z⇒w z zw ∖ z ⇒ z, and it is a contradiction. (b) By the above result (a), ℓ¯w⊤(ℓ¯z1−ℓ¯w)=ℓ¯w⊤(ℓ¯z0−ℓ¯w)=0superscriptsubscript¯ℓtopsubscript¯ℓsubscript1subscript¯ℓsuperscriptsubscript¯ℓtopsubscript¯ℓsubscript0subscript¯ℓ0 _w ( _z_1- _w)= _w % ( _z_0- _w)=0over¯ start_ARG ℓ end_ARGw⊤ ( over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw ) = over¯ start_ARG ℓ end_ARGw⊤ ( over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw ) = 0. Therefore, ℓ¯w⊤(ℓ¯z1−ℓ¯z0)=0superscriptsubscript¯ℓtopsubscript¯ℓsubscript1subscript¯ℓsubscript00 _w ( _z_1- _z_0)=0over¯ start_ARG ℓ end_ARGw⊤ ( over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) = 0. (c) Let’s say that w1subscript1w_1w1 is wZsubscriptw_Zwitalic_Z defined in Definition 2. The binary contrast z0⇒z1⇒subscript0subscript1z_0 z_1z0 ⇒ z1 is subordinate to the binary feature for the attribute w0subscript0w_0w0. By the property of the causal inner product, ℓ¯w0subscript¯ℓsubscript0 _w_0over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT is orthogonal to the linear representation ℓ¯z0⇒z1=ℓ¯z1−ℓ¯z0subscript¯ℓ⇒subscript0subscript1subscript¯ℓsubscript1subscript¯ℓsubscript0 _z_0 z_1= _z_1- _z_0over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 0 ⇒ z1 end_POSTSUBSCRIPT = over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT (by Corollary 10). Then, with the above result (b), we have (ℓ¯w1−ℓ¯w0)⊤(ℓ¯z1−ℓ¯z0)superscriptsubscript¯ℓsubscript1subscript¯ℓsubscript0topsubscript¯ℓsubscript1subscript¯ℓsubscript0( _w_1- _w_0) ( _z_1- _% z_0)( over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT )⊤ ( over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGz start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ). (d) By the above result (a), we have ‖ℓ¯w1−ℓ¯w0‖22=‖ℓ¯w1‖22−‖ℓ¯w0‖22‖ℓ¯w2−ℓ¯w1‖22=‖ℓ¯w2‖22−‖ℓ¯w1‖22‖ℓ¯w2−ℓ¯w0‖22=‖ℓ¯w2‖22−‖ℓ¯w0‖22.casessuperscriptsubscriptnormsubscript¯ℓsubscript1subscript¯ℓsubscript022superscriptsubscriptnormsubscript¯ℓsubscript122superscriptsubscriptnormsubscript¯ℓsubscript022otherwisesuperscriptsubscriptnormsubscript¯ℓsubscript2subscript¯ℓsubscript122superscriptsubscriptnormsubscript¯ℓsubscript222superscriptsubscriptnormsubscript¯ℓsubscript122otherwisesuperscriptsubscriptnormsubscript¯ℓsubscript2subscript¯ℓsubscript022superscriptsubscriptnormsubscript¯ℓsubscript222superscriptsubscriptnormsubscript¯ℓsubscript022otherwise cases\| _w_1- _w_0\|_2^2=\| _w% _1\|_2^2-\| _w_0\|_2^2\\ \| _w_2- _w_1\|_2^2=\| _w_2\|_2^% 2-\| _w_1\|_2^2\\ \| _w_2- _w_0\|_2^2=\| _w_2\|_2^% 2-\| _w_0\|_2^2. cases start_ROW start_CELL ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥22 = ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∥22 - ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥22 end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∥22 = ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∥22 - ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∥22 end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥22 = ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∥22 - ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥22 . end_CELL start_CELL end_CELL end_ROW (B.9) Then, ‖ℓ¯w1−ℓ¯w0‖22+‖ℓ¯w2−ℓ¯w1‖22superscriptsubscriptnormsubscript¯ℓsubscript1subscript¯ℓsubscript022superscriptsubscriptnormsubscript¯ℓsubscript2subscript¯ℓsubscript122 \| _w_1- _w_0\|_2^2+\| _w% _2- _w_1\|_2^2∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥22 + ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∥22 (B.10) =‖ℓ¯w1‖22−‖ℓ¯w0‖22+‖ℓ¯w2‖22−‖ℓ¯w1‖22absentsuperscriptsubscriptnormsubscript¯ℓsubscript122superscriptsubscriptnormsubscript¯ℓsubscript022superscriptsubscriptnormsubscript¯ℓsubscript222superscriptsubscriptnormsubscript¯ℓsubscript122 =\| _w_1\|_2^2-\| _w_0\|_2^2+\|% _w_2\|_2^2-\| _w_1\|_2^2= ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∥22 - ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥22 + ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∥22 - ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ∥22 (B.11) =‖ℓ¯w2‖22−‖ℓ¯w0‖22absentsuperscriptsubscriptnormsubscript¯ℓsubscript222superscriptsubscriptnormsubscript¯ℓsubscript022 =\| _w_2\|_2^2-\| _w_0\|_2^2= ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ∥22 - ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥22 (B.12) =‖ℓ¯w2−ℓ¯w0‖22.absentsuperscriptsubscriptnormsubscript¯ℓsubscript2subscript¯ℓsubscript022 =\| _w_2- _w_0\|_2^2.= ∥ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ∥22 . (B.13) Therefore, ℓ¯w1−ℓ¯w0subscript¯ℓsubscript1subscript¯ℓsubscript0 _w_1- _w_0over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT is orthogonal to ℓ¯w2−ℓ¯w1subscript¯ℓsubscript2subscript¯ℓsubscript1 _w_2- _w_1over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT. ∎ Appendix C Natural Categorical Concepts as Simplices Polytopes are quite general objects. Definition 7 also includes representations of categorical variables that are semantically unnatural, e.g., dog,sandwich,runningdogsandwichrunning\ dog, sandwich, running\ dog , sandwich , running . We would like to make a more precise statement about the representation of “natural” concepts. One possible notion of a “natural” concept is one where the model can freely manipulate the output values. The next proposition shows such concepts have particularly simple structure: Proposition 9 (Categorical Concepts are Represented as Simplices). Suppose that w0,…,wk−1subscript0…subscript1\w_0,…,w_k-1\ w0 , … , witalic_k - 1 is a collection of k mutually exclusive attributes such that for every joint distribution Q(w0,…wk−1)subscript0…subscript1Q(w_0,… w_k-1)Q ( w0 , … witalic_k - 1 ) there is some ℓisubscriptℓ _iℓitalic_i such that ℙ(W=wi|ℓi)=Q(W=wi)ℙconditionalsubscriptsubscriptℓsubscriptP(W=w_i~|~ _i)=Q(W=w_i)blackboard_P ( W = witalic_i | ℓitalic_i ) = Q ( W = witalic_i ) for every i. Then, the vector representations ℓ¯w0,…,ℓ¯wk−1subscript¯ℓsubscript0…subscript¯ℓsubscript1 _w_0,…, _w_k-1over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , … , over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT k - 1 end_POSTSUBSCRIPT form a (k−1)1(k-1)( k - 1 )-simplex in the representation space. In this case, we take the simplex to be the representation of the categorical concept W=w0,…,wk−1subscript0…subscript1W=\w_0,…,w_k-1\W = w0 , … , witalic_k - 1 . Figure 7: Illustration of the case k=33k=3k = 3 in the proof of Proposition 9. Proof. If we can represent arbitrary joint distributions, this means, in particular, that we can change the probability of one attribute without changing the relative probability between a pair of other attributes. Consider the case where k=33k=3k = 3, as illustrated in Figure 7. If ℓ¯w0,ℓ¯w1,ℓ¯w2subscript¯ℓsubscript0subscript¯ℓsubscript1subscript¯ℓsubscript2 _w_0, _w_1, _w_2over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT are on a line, then there is no direction in that line (to change the value in the categorical concept) such that adding the direction can change the probability of w2subscript2w_2w2 without changing the relative probabilities between w0subscript0w_0w0 and w1subscript1w_1w1. However, if ℓ¯w0,ℓ¯w1,ℓ¯w2subscript¯ℓsubscript0subscript¯ℓsubscript1subscript¯ℓsubscript2 _w_0, _w_1, _w_2over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT are not on a line, they form a triangle. Then, there exists a line that is toward ℓ¯w2subscript¯ℓsubscript2 _w_2over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT and perpendicular to the opposite side of the triangle. Now adding the direction ℓ~~ℓ over~ start_ARG ℓ end_ARG can manipulate the probability of w2subscript2w_2w2 without changing the relative probabilities between w0subscript0w_0w0 and w1subscript1w_1w1. That is, for any α>00α>0α > 0 and context embedding ℓ ℓ, ℙ(W=w2|ℓ+αℓ~)>ℙ(W=w2|ℓ), andℙ(W=w1|ℓ+αℓ~)ℙ(W=w0|ℓ+αℓ~)=ℙ(W=w1|ℓ)ℙ(W=w0|ℓ).casesℙconditionalsubscript2ℓ~ℓℙconditionalsubscript2ℓ andotherwiseℙconditionalsubscript1ℓ~ℓℙconditionalsubscript0ℓ~ℓℙconditionalsubscript1ℓℙconditionalsubscript0ℓotherwise casesP(W=w_2~|~ +α )>P(W=w_% 2~|~ ), and\\ P(W=w_1~|~ +α )P(W=w_0~% |~ +α )= P(W=w_1~|~ )P% (W=w_0~|~ ). cases start_ROW start_CELL blackboard_P ( W = w2 | ℓ + α over~ start_ARG ℓ end_ARG ) > blackboard_P ( W = w2 | ℓ ) , and end_CELL start_CELL end_CELL end_ROW start_ROW start_CELL divide start_ARG blackboard_P ( W = w1 | ℓ + α over~ start_ARG ℓ end_ARG ) end_ARG start_ARG blackboard_P ( W = w0 | ℓ + α over~ start_ARG ℓ end_ARG ) end_ARG = divide start_ARG blackboard_P ( W = w1 | ℓ ) end_ARG start_ARG blackboard_P ( W = w0 | ℓ ) end_ARG . end_CELL start_CELL end_CELL end_ROW (C.1) Therefore, the vectors ℓ¯w0,ℓ¯w1,ℓ¯w2subscript¯ℓsubscript0subscript¯ℓsubscript1subscript¯ℓsubscript2 _w_0, _w_1, _w_2over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT form a 2-simplex. This argument extends immediately to higher k by induction. For each i∈0,…,k−10…1i∈\0,…,k-1\i ∈ 0 , … , k - 1 , there should exist a direction that is toward ℓ¯wisubscript¯ℓsubscript _w_iover¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT i end_POSTSUBSCRIPT and orthogonal to the opposite hyperplane ((k−2)2(k-2)( k - 2 )-simplex) formed by the other ℓ¯wi′subscript¯ℓsubscriptsuperscript′ _w_i over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT i′ end_POSTSUBSCRIPT’s. Then, the vectors ℓ¯w0,…,ℓ¯wk−1subscript¯ℓsubscript0…subscript¯ℓsubscript1 _w_0,…, _w_k-1over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , … , over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT k - 1 end_POSTSUBSCRIPT form a (k−1)1(k-1)( k - 1 )-simplex. ∎ Appendix D Subspaces for Categorical Concepts By Theorem 4, polytope representations have the following property: Corollary 10 (Polytope Representations). Let W=w0,…,wk−1subscript0…subscript1W=\w_0,…,w_k-1\W = w0 , … , witalic_k - 1 be a categorical concept, and suppose there exist vector representations ℓ¯wisubscript¯ℓsubscript _w_iover¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT i end_POSTSUBSCRIPT for the binary features of each attribute wisubscriptw_iwitalic_i (their convex hull is the polytope representation of W). Let L¯=[ℓ¯w1−ℓ¯w0,…,ℓ¯wk−1−ℓ¯w0]∈ℝd×(k−1)¯subscript¯ℓsubscript1subscript¯ℓsubscript0…subscript¯ℓsubscript1subscript¯ℓsubscript0superscriptℝ1 L= [ _w_1- _w_0,…, _w_k-1% - _w_0 ] ^d×(k-1)over¯ start_ARG L end_ARG = [ over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , … , over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT k - 1 end_POSTSUBSCRIPT - over¯ start_ARG ℓ end_ARGw start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ] ∈ blackboard_Rd × ( k - 1 ) and let (L¯)¯C( L)C ( over¯ start_ARG L end_ARG ) be its column space. Then, there exist some vectors τwi∈(L¯)subscriptsubscript¯ _w_i ( L)τitalic_w start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ∈ C ( over¯ start_ARG L end_ARG ), for each i, such that ∏L¯g(y)=τwiif y∈(wi)∏L¯g(y)=0if y∉∪i=0k−1(wi),casessubscriptproduct¯subscriptsubscriptif subscriptsubscriptproduct¯0if superscriptsubscript01subscript cases _ Lg(y)= _w_i&if y (w_i)% \\ _ Lg(y)=0&if y ∈ _i=0^k-1Y(w_i),% cases start_ROW start_CELL ∏over¯ start_ARG L end_ARG g ( y ) = τitalic_w start_POSTSUBSCRIPT i end_POSTSUBSCRIPT end_CELL start_CELL if y ∈ Y ( witalic_i ) end_CELL end_ROW start_ROW start_CELL ∏over¯ start_ARG L end_ARG g ( y ) = 0 end_CELL start_CELL if y ∉ ∪i = 0k - 1 Y ( witalic_i ) , end_CELL end_ROW (D.1) where ∏L¯=L¯(L¯⊤L¯)†L¯⊤subscriptproduct¯superscriptsuperscript¯top¯†superscript¯top _ L= L( L L) L ∏over¯ start_ARG L end_ARG = over¯ start_ARG L end_ARG ( over¯ start_ARG L end_ARG⊤ over¯ start_ARG L end_ARG )† over¯ start_ARG L end_ARG⊤ is the projection matrix onto (L¯)¯C( L)C ( over¯ start_ARG L end_ARG ), and (L¯⊤L¯)†superscriptsuperscript¯top¯†( L L) ( over¯ start_ARG L end_ARG⊤ over¯ start_ARG L end_ARG )† denotes a pseudo-inverse of the matrix. Proof. By Theorem 4, for i=1,…,k−11…1i=1,…,k-1i = 1 , … , k - 1, we have ∏L¯g(y)=L¯(L¯⊤L¯)†L¯⊤g(y)=L¯(L¯⊤L¯)†[0,…,0,bwi2,0,…,0]⊤:=τwisubscriptproduct¯superscriptsuperscript¯top¯†superscript¯top¯superscriptsuperscript¯top¯†superscript0…0superscriptsubscriptsubscript20…0topassignsubscriptsubscript _ Lg(y)= L( L L) L g(y)% = L( L L) [0,…,0,b_w_i^2,0,…% ,0 ] := _w_i∏over¯ start_ARG L end_ARG g ( y ) = over¯ start_ARG L end_ARG ( over¯ start_ARG L end_ARG⊤ over¯ start_ARG L end_ARG )† over¯ start_ARG L end_ARG⊤ g ( y ) = over¯ start_ARG L end_ARG ( over¯ start_ARG L end_ARG⊤ over¯ start_ARG L end_ARG )† [ 0 , … , 0 , bitalic_w start_POSTSUBSCRIPT i end_POSTSUBSCRIPT2 , 0 , … , 0 ]⊤ := τitalic_w start_POSTSUBSCRIPT i end_POSTSUBSCRIPT (D.2) for any y∈(wi)subscripty (w_i)y ∈ Y ( witalic_i ). Similarly, we have ∏L¯g(y)=L¯(L¯⊤L¯)†L¯⊤g(y)=L¯(L¯⊤L¯)†[−bw02,…,−bw02]⊤:=τw0subscriptproduct¯superscriptsuperscript¯top¯†superscript¯top¯superscriptsuperscript¯top¯†superscriptsuperscriptsubscriptsubscript02…superscriptsubscriptsubscript02topassignsubscriptsubscript0 _ Lg(y)= L( L L) L g(y)% = L( L L) [-b_w_0^2,…,-b_w_0% ^2 ] := _w_0∏over¯ start_ARG L end_ARG g ( y ) = over¯ start_ARG L end_ARG ( over¯ start_ARG L end_ARG⊤ over¯ start_ARG L end_ARG )† over¯ start_ARG L end_ARG⊤ g ( y ) = over¯ start_ARG L end_ARG ( over¯ start_ARG L end_ARG⊤ over¯ start_ARG L end_ARG )† [ - bitalic_w start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT2 , … , - bitalic_w start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT2 ]⊤ := τitalic_w start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT (D.3) for any y∈(w0)subscript0y (w_0)y ∈ Y ( w0 ), while ∏L¯g(y)=L¯(L¯⊤L¯)†L¯⊤g(y)=L¯(L¯⊤L¯)†[0,…,0]⊤=0subscriptproduct¯superscriptsuperscript¯top¯†superscript¯top¯superscriptsuperscript¯top¯†superscript0…0top0 _ Lg(y)= L( L L) L g(y)% = L( L L) [0,…,0 ] =0∏over¯ start_ARG L end_ARG g ( y ) = over¯ start_ARG L end_ARG ( over¯ start_ARG L end_ARG⊤ over¯ start_ARG L end_ARG )† over¯ start_ARG L end_ARG⊤ g ( y ) = over¯ start_ARG L end_ARG ( over¯ start_ARG L end_ARG⊤ over¯ start_ARG L end_ARG )† [ 0 , … , 0 ]⊤ = 0 (D.4) for any y∉∪i=0k−1(wi)superscriptsubscript01subscripty ∈ _i=0^k-1Y(w_i)y ∉ ∪i = 0k - 1 Y ( witalic_i ). Therefore, we have eq. D.1 with some constant vectors τwisubscriptsubscript _w_iτitalic_w start_POSTSUBSCRIPT i end_POSTSUBSCRIPT. ∎ In words, the unembeddings of tokens for the same attribute share the same projections on the subspace spanned by the differences between the vector representations. Therefore, adding any vector in the subspace (L¯)¯C( L)C ( over¯ start_ARG L end_ARG ) to the context embedding changes the probability of the target concept W=w0,…,wk−1subscript0…subscript1W=\w_0,…,w_k-1\W = w0 , … , witalic_k - 1 without changing any other concept subordinate to or causally separable with the target concept. Finally, note that Corollary 10 implies Corollary 6. Appendix E Experiment Details We employ the Gemma-2B version of the Gemma model (Mesnard et al., 2024), which is accessible online via the huggingface library. Its two billion parameters are pre-trained on three trillion tokens. This model utilizes 256K tokens and 2,048 dimensions for the representation space. We always use tokens that start with a space (‘\\ 2581’) in front of the word, as they are used for next-word generation with full meaning. Additionally, like WordNet data we use, we include plural forms, and both capital and lowercase versions of the words in (animal)animalY( animal)Y ( animal ) and (plant)plantY( plant)Y ( plant ) for visualization in Appendix A. In the WordNet synset data, each content of the synset mammal.n.01 indicates that ”mammal” is a word, ”n” denotes ”noun,” and ”01” signifies the first meaning of the word. In the WordNet hierarchy, if a parent has only one child, we combine the two features into one. Additionally, since the WordNet hierarchy is not a perfect tree, a child can have more than one parent. We use one of the parents when computing the ℓ¯w−ℓ¯parent of wsubscript¯ℓsubscript¯ℓparent of _w- _parent of wover¯ start_ARG ℓ end_ARGw - over¯ start_ARG ℓ end_ARGparent of w. Lastly, we use the vector representations estimated by the non-split collection of tokens for Figure 2, Table 1, Figure 4, Figure 6, Figure 14, and Figure 18. Appendix F Additional Results Zooming in on a Subtree of Noun Hierarchy Figure 8: Subtree in WordNet noun hierarchy for descendants of animal. Figure 9: Zoomed-in Heatmaps of the subtree for animal in Figure 8. As it is difficult to understand the entire WordNet hierarchy at once from the heatmaps in Figure 4, we present a zoomed-in heatmap for the subtree (Figure 8) for the feature animal in Figure 9. The left heatmap displays the shortest distance between the nodes of the subtree in Figure 8. The middle heatmap shows that the cosine similarities between the vector representations ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw reflect the child-parent or sibling relationships. The right heatmap demonstrates that the vector representations estimated by shuffled unembeddings only reflect the set inclusion relationships, and other pairs have cosine similarities close to 0. Figure 10: WordNet noun hierarchy is encoded as the orthogonal structure predicted by statement (d) in Theorem 8. The cosine similarity between a child-parent vector and a parent-grandparent vector for each feature in the hierarchy estimated by original (blue) and shuffled (green) unembedding vectors with the whole (left) and training (right) set of tokens. Another baseline (orange) is the cosine similarity with random parent and grandparent vectors. Additional Results on Hierarchical Orthogonality Analogous to Figure 5, we validate the statement (d) in Theorem 8. Figure 10 shows that a child-parent vector and a parent-grandparent vector for each feature estimated by the original unembedidngs (blue) are orthogonal. The random parent and grandparent vectors (orange) make the cosine similarity not close to 0, suggesting that the orthogonality is not merely a byproduct of the high-dimensional space. Similar to the findings in Figure 5, the shuffled unembedding vectors (green) reflect the set inclusion relationships, and the cosine similarities are close to 0. When we violate the set inclusion by using only the 70% training set of tokens, the right plot in Figure 10 shows that the cosine similarities from shuffled (green) unembeddings are much smaller, while those from original (blue) unembeddings are still close to 0. This indicates that the hierarchical orthogonality is not a trivial consequence of the set inclusion, but rather reflects the semantic hierarchy. Figure 11: Comparison of projection of train (green), test (blue), and random (orange) words on estimated mean vector for each WordNet feature, from the original unembeddings (left) and the shuffled unembeddings (right). The value for each word y is (g(y)⊤ℓ¯w)/‖ℓ¯w‖22superscripttopsubscript¯ℓsuperscriptsubscriptnormsubscript¯ℓ22(g(y) _w)/\| _w\|_2^2( g ( y )⊤ over¯ start_ARG ℓ end_ARGw ) / ∥ over¯ start_ARG ℓ end_ARGw ∥22 that we expect to be 1 when y has the target feature. The x-axis indices denote all features in the noun hierarchy. The thick lines present the mean of the projections for each feature and the error bars indicate the standard deviation. The features are ordered by the hierarchy. The Mean Estimator for the Vector Representation Has High Variance The mean vector (gw)subscriptE(g_w)blackboard_E ( gitalic_w ) can serve as an estimator for the vector representation ℓ¯wsubscript¯ℓ _wover¯ start_ARG ℓ end_ARGw. However, Figure 11, which is analogous to Figure 3, shows that the projections of train and test words onto the mean vector have larger variances. Therefore, the LDA direction (eq. 5.2) would be a more appropriate estimator for the vector representation. Figure 12: WordNet noun hierarchy is not encoded as the orthogonal structure when we use the naive Euclidean inner product, whereas it is encoded as orthogonality in the causal inner product. Geometry of Hierarchical relations on the Euclidean Inner Product In this paper, we transform the representation spaces to use the causal inner product. However, what if we employ the naive Euclidean inner product instead? To address this, we estimate the vector representations for the WordNet noun hierarchy estimated from the original Gemma unembedding vectors through centering alone, without applying whitening (which is necessary for the causal inner product). This approach still preserves the Euclidean inner product. Then, Figure 12 shows that the hierarchical orthogonality is not satisfied in the Euclidean inner product space, as evidenced by the cosine similarities not being close to 0. Figure 13: Vector representations exist for most binary features in the WordNet verb hierarchy. Figure 14: WordNet verb hierarchy is encoded in Gemma representation space. Figure 15: WordNet verb hierarchy is encoded as the orthogonal structure predicted by statement (a) in Theorem 8. Figure 16: WordNet verb hierarchy is encoded as the orthogonal structure predicted by statement (d) in Theorem 8. WordNet Verb Hierarchy In the same way as for the noun hierarchy, we estimate the vector representations for the WordNet verb hierarchy. Analogous to Figure 3, Figure 13 shows that the vector representations exist for most binary features in the WordNet verb hierarchy. Analogous to Figure 4, Figure 14 shows that the hierarchical semantics in WordNet are encoded in the Gemma representation space. Analogous to Figure 5 and Figure 10, Figure 15 and Figure 16 show that the hierarchical orthogonality is encoded in the Gemma representation space for the WordNet verb hierarchy. Figure 17: Vector representations for most binary features in the WordNet noun hierarchy exist in the LLaMA-3 model. Figure 18: WordNet noun hierarchy is encoded in LLaMA-3 representation space. Figure 19: In LLaMA-3 representation space, WordNet noun hierarchy is encoded as the orthogonal structure predicted by statement (a) in Theorem 8. Figure 20: In LLaMA-3 representation space, WordNet noun hierarchy is encoded as the orthogonal structure predicted by statement (d) in Theorem 8. WordNet Noun Hierarchy encoded in LLaMA-3 Model We also validate the theoretical predictions in the LLaMA-3-8B model (Dubey et al., 2024) in the same way as for the Gemma model. The vector representations for most binary features in the WordNet noun hierarchy exist in the LLaMA-3 model (Figure 17). The hierarchical semantics in WordNet are encoded in the LLaMA-3 representation space (Figure 18). Lastly, the hierarchical orthogonality is encoded in the LLaMA-3 representation space for the WordNet noun hierarchy (Figure 19 and Figure 20). Appendix G Why does set inclusion give the orthogonality? The left panel of Figure 5 shows that the cosine similarity between the child-parent vector and the parent vector is also close to 0 when they are estimated by the shuffled unembedding vectors. One of the possible explanations for this phenomenon is that the set inclusion relationships (and the estimating process) give the orthogonality. For example, the set inclusion between the collections (z)=dog,sandwich,runningdogsandwichrunningY(z)=\ dog, sandwich, running\Y ( z ) = dog , sandwich , running for a child z and (w)=dog,sandwich,running,France,scientist,diamonddogsandwichrunningFrancescientistdiamondY(w)=\ dog, sandwich, running, % France, scientist, diamond\Y ( w ) = dog , sandwich , running , France , scientist , diamond for a parent w can make the child-parent vector and the parent vector orthogonal. Since the LDA-based vector representation estimated by eq. 5.2 is similar to the mean vector (gw)subscriptE(g_w)blackboard_E ( gitalic_w ), we briefly derive the orthogonality using the mean vector as the vector representation. Suppose that a1,…,aNasubscript1…subscriptsubscripta_1,…,a_N_aa1 , … , aitalic_N start_POSTSUBSCRIPT a end_POSTSUBSCRIPT, b1,…,bNb∼i.i.d.(0,Id)b_1,…,b_N_b i.i.d. N(0,I_d)b1 , … , bitalic_N start_POSTSUBSCRIPT b end_POSTSUBSCRIPT start_OVERACCENT i . i . d . end_OVERACCENT start_ARG ∼ end_ARG N ( 0 , Iitalic_d ), which may correspond to the (centered and whitened) shuffled unembeddings. Here, aisubscripta_iaitalic_i’s are for the child z, and both aisubscripta_iaitalic_i’s and bisubscriptb_ibitalic_i’s are for the parent w. Then, the parent vector is vw=1Na+Nb(∑ai+∑bi)subscript1subscriptsubscriptsubscriptsubscriptv_w= 1N_a+N_b(Σ a_i+Σ b_i)vitalic_w = divide start_ARG 1 end_ARG start_ARG Nitalic_a + Nitalic_b end_ARG ( ∑ aitalic_i + ∑ bitalic_i ), and the child vector is vz=1Na∑aisubscript1subscriptsubscriptv_z= 1N_aΣ a_ivitalic_z = divide start_ARG 1 end_ARG start_ARG Nitalic_a end_ARG ∑ aitalic_i. We know that a1,…,aNasubscript1…subscriptsubscripta_1,…,a_N_aa1 , … , aitalic_N start_POSTSUBSCRIPT a end_POSTSUBSCRIPT, b1,…,bNbsubscript1…subscriptsubscriptb_1,…,b_N_bb1 , … , bitalic_N start_POSTSUBSCRIPT b end_POSTSUBSCRIPT are mutually orthogonal and ‖ai‖2=‖bi‖2=dsuperscriptnormsubscript2superscriptnormsubscript2\|a_i\|^2=\|b_i\|^2=d∥ aitalic_i ∥2 = ∥ bitalic_i ∥2 = d with high probability. Therefore, we have (vz−vw)⊤vw=superscriptsubscriptsubscripttopsubscriptabsent (v_z-v_w) v_w=( vitalic_z - vitalic_w )⊤ vitalic_w = (G.1) =(NbNa(Na+Nb)∑ai−1Na+Nb∑bi)⊤(1Na+Nb∑ai+1Na+Nb∑bi)absentsuperscriptsubscriptsubscriptsubscriptsubscriptsubscript1subscriptsubscriptsubscripttop1subscriptsubscriptsubscript1subscriptsubscriptsubscript = ( N_bN_a(N_a+N_b)Σ a_i- 1N_a+% N_bΣ b_i ) ( 1N_a+N_bΣ a_i+ 1N% _a+N_bΣ b_i )= ( divide start_ARG Nitalic_b end_ARG start_ARG Nitalic_a ( Nitalic_a + Nitalic_b ) end_ARG ∑ aitalic_i - divide start_ARG 1 end_ARG start_ARG Nitalic_a + Nitalic_b end_ARG ∑ bitalic_i )⊤ ( divide start_ARG 1 end_ARG start_ARG Nitalic_a + Nitalic_b end_ARG ∑ aitalic_i + divide start_ARG 1 end_ARG start_ARG Nitalic_a + Nitalic_b end_ARG ∑ bitalic_i ) (G.2) =NbNa(Na+Nb)2(∑ai)⊤∑ai−1(Na+Nb)2(∑bi)⊤∑biabsentsubscriptsubscriptsuperscriptsubscriptsubscript2superscriptsubscripttopsubscript1superscriptsubscriptsubscript2superscriptsubscripttopsubscript = N_bN_a(N_a+N_b)^2(Σ a_i) Σ a_i% - 1(N_a+N_b)^2(Σ b_i) Σ b_i= divide start_ARG Nitalic_b end_ARG start_ARG Nitalic_a ( Nitalic_a + Nitalic_b )2 end_ARG ( ∑ aitalic_i )⊤ ∑ aitalic_i - divide start_ARG 1 end_ARG start_ARG ( Nitalic_a + Nitalic_b )2 end_ARG ( ∑ bitalic_i )⊤ ∑ bitalic_i (G.3) =NbNa(Na+Nb)2(Nad)−1(Na+Nb)2(Nbd)absentsubscriptsubscriptsuperscriptsubscriptsubscript2subscript1superscriptsubscriptsubscript2subscript = N_bN_a(N_a+N_b)^2(N_ad)- 1(N_a+N_% b)^2(N_bd)= divide start_ARG Nitalic_b end_ARG start_ARG Nitalic_a ( Nitalic_a + Nitalic_b )2 end_ARG ( Nitalic_a d ) - divide start_ARG 1 end_ARG start_ARG ( Nitalic_a + Nitalic_b )2 end_ARG ( Nitalic_b d ) (G.4) =0absent0 =0= 0 (G.5) with high probability. In words, the orthogonality between the child-parent vector and the parent vector can be derived from the set inclusion relationships. We can say that the set inclusion is the hierarchical relation as defined in Definition 2, but natural semantic hierarchy is our main focus in this paper. In this context, the right panel of Figure 5 shows that the violation of the set inclusion relationships makes the cosine similarity not close to 0, suggesting that semantic hierarchy is encoded as orthogonality.