Paper deep dive
Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation
Pedro H. V. Valois, Lincon S. Souza, Erica K. Shimomoto, Kazuhiro Fukui
Models: Gemma 2, Llama 3.1 8B, Phi 3
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 5:31:57 PM
Summary
The paper introduces the Frame Representation Hypothesis (FRH), an extension of the Linear Representation Hypothesis (LRH) that models multi-token words as 'frames' (ordered sequences of vectors) to improve LLM interpretability and control. It proposes 'Concept Frames' as centroids of word frames and demonstrates 'Top-k Concept-Guided Decoding' to steer text generation, validated on Llama 3.1, Gemma 2, and Phi 3 models.
Entities (5)
Relation Signals (3)
Frame Representation Hypothesis → extends → Linear Representation Hypothesis
confidence 100% · FRH formally extends the LRH to multi-token words.
Top-k Concept-Guided Decoding → implements → Frame Representation Hypothesis
confidence 95% · this algorithm aligns model outputs with desired concepts, a practical prototype for FRH
Open Multilingual WordNet → providesdatafor → Frame Representation Hypothesis
confidence 90% · we leverage the Open Multilingual WordNet (OMW) as a source of synonyms to build concepts
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Interpretability is a key challenge in fostering trust for Large Language Models (LLMs), which stems from the complexity of extracting reasoning from model's parameters. We present the Frame Representation Hypothesis, a theoretically robust framework grounded in the Linear Representation Hypothesis (LRH) to interpret and control LLMs by modeling multi-token words. Prior research explored LRH to connect LLM representations with linguistic concepts, but was limited to single token analysis. As most words are composed of several tokens, we extend LRH to multi-token words, thereby enabling usage on any textual data with thousands of concepts. To this end, we propose words can be interpreted as frames, ordered sequences of vectors that better capture token-word relationships. Then, concepts can be represented as the average of word frames sharing a common concept. We showcase these tools through Top-k Concept-Guided Decoding, which can intuitively steer text generation using concepts of choice. We verify said ideas on Llama 3.1, Gemma 2, and Phi 3 families, demonstrating gender and language biases, exposing harmful content, but also potential to remediate them, leading to safer and more transparent LLMs. Code is available at this https URL
Tags
Links
Trouble viewing inline? Open PDF directly →
Full Text
53,774 characters extracted from source content.
Expand or collapse full text
Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation Pedro H. V. Valois ∗ , Lincon S. Souza † , Erica K. Shimomoto † , Kazuhiro Fukui ∗ ∗ University of Tsukuba, † National Institute of Advanced Industrial Science and Technology (AIST) Abstract Interpretability is a key challenge in fos- tering trust for Large Language Models (LLMs), which stems from the complexity of extracting reasoning from model’s param- eters. We present the Frame Representa- tion Hypothesis, a theoretically robust frame- work grounded in the Linear Representation Hypothesis (LRH) to interpret and control LLMs by modeling multi-token words. Prior research explored LRH to connect LLM rep- resentations with linguistic concepts, but was limited to single token analysis. As most words are composed of several tokens, we extend LRH to multi-token words, thereby enabling usage on any textual data with thou- sands of concepts. To this end, we pro- pose words can be interpreted as frames, or- dered sequences of vectors that better cap- ture token-word relationships. Then, con- cepts can be represented as the average of word frames sharing a common concept. We showcase these tools through Top-kConcept- Guided Decoding, which can intuitively steer text generation using concepts of choice. We verify said ideas on Llama 3.1, Gemma 2, and Phi 3 families, demonstrating gender and language biases, exposing harmful content, but also potential to remediate them, leading to safer and more transparent LLMs. Code is available at https://github.com/phvv-me/ frame-representation-hypothesis.git 1 Introduction Interpretability in deep learning aims to elucidate how neural networks derive predictions. As models grow complex, understanding internal mechanisms gets challenging. By identifying factors contribut- ing to the output, we can foster trust, safety, fairness and improve capabilities (Hooker et al., 2019). The goal of this study is to enhance the inter- pretability and control of LLMs via the encoding of human-comprehensiblelinguistic concepts. LLMs represent text through tokens, which can be a word, part of a word, or even a character, as per models’ design. In contrast, humans better understand text throughconcepts, cognitive symbols that depict reality, often grouping related objects, events, or further abstractions based on shared characteristics. Our purpose is to provide tools to represent con- cepts within LLMs, allowing output explanations that are suited for our mental models. Prominent works that offer such tools are based on the Linear Representation Hypothesis (LRH): it suggests linear operations on token vectors can ex- plain model behavior, with concepts represented as vectors in LLM feature space (Templeton et al., 2024). For instance, we can identify the con- ceptfemaleas the average of token vectors like f(woman)orf(queen). Thus, token vectors en- code more than just lexical data: they also represent linguistic concepts (Mikolov et al., 2013). Nevertheless, LRH’s concepts are 1-dimensional, constraining them to single-token words, which are a minor fraction of any given language (Bau et al., 2020). As exemplified in Figure 1, a concept like vegetarianis exclusively linked to multi-token words, such asmeatlessorherbivore, meaning multiple vectors are required to represent it. Since most words are constituted of several tokens, 1- dimensional structures prevent LRH application in most interpretability tasks effectively. To address this shortcoming of LRH, we propose a new framework for LLM interpretability, based on a key empirical observation about the nature of LLMs: our experiments show that over 99% of words among several languages are composed of linearly independent token vectors. This allows us to address multi-token words by proposing the Frame Representation Hypothesis (FRH), which as- sumes words are ordered sequences of independent vectors – mathematically identified as frames. Starting from this postulate, we develop a mathe- matical framework to represent words and concepts arXiv:2412.07334v2 [cs.CL] 12 Dec 2024 Frame Concept meatless animal vegetarianism meat herbivore vegetarian carnivore meatball Word (Frame) 푢(푣푒푔) 푢(etarian) Semantic Frame Space ڂ 푖=1 푘 푺풕(푖,푑) Token Space ℝ 푑 푢(ism) Embedding Figure 1:Frame Representation Hypothesis Overview: Tokens are vectors, which combine into words as multi-dimensional frames. In turn, Con- cept Frames are centroids of word sets. as frames; we define aSemantic Frame Space, and equip it with a correlation between frames that preserves the token whitening mechanism intro- duced by LRH (Park et al., 2023), allowing us to measure frames semantic relationship. Following Figure 1, the frame representation lets us identify words, such asmeatless,meatandmeatballas distinct geometrical objects, although they have to- kens in common. Then, we can compute Concept Frames likevegetarianorcarnivoreas the centroid for a set of words sharing that concept. Furthermore, we introduce Top-kConcept- Guided Decoding (Figure 2), which controls text generation by selecting the tokens which maxi- mize a chosen concept. For example, the concept vegetarianwould guide inputI liketoI like fruitsif the top-3options werebeef,footballor fruits. This algorithm aligns model outputs with desired concepts, a practical prototype for FRH that allows meaningful LLM understanding. To that end, we leverage the Open Multilingual WordNet (OMW) (Bond and Foster, 2013) as a source of synonyms to build concepts. We use over 50M words among multiple languages to build over 100,000 Concept Frames, enabling rich model understanding in a diverse yet inexpensive manner. In short, FRH formally extends the LRH to multi- token words. We show its validity both from the theoretical and empirical points of view. Our primary contributions are as follows: 1. Frame Representation Hypothesis as an exten- sion of LRH to multi-token words by defining them as Frames, thereby addressing the limi- tations of single-token representations. 2. Proposal of Concept Frames to represent lin- guistic concepts from a set of Word Frames. 3. Development of Top-kConcept-Guided De- coding, a proof-of-concept application to steer text generation using chosen concepts and ex- pose model biases or potential vulnerabilities. 2 Related Work We briefly review LLM interpretability, control- lable text generation and Frame usage in the field. Language Models InterpretabilityThe widespread adoption of LLMs brought attention to the need of understanding their inner-workings, risks, and limitations. Several studies identified a common property to these models that became known as the Linear Representation Hypothesis, encoding model knowledge as vectors (Mikolov et al., 2013), and enabling model explanation and editing (Wang et al., 2023). Also, the Super- position Hypothesis (SH) assumes specialized information is superimposed in LLM feature spaces. These ideas underpin Sparse Autoencoders, which learn dictionaries of interpretable concepts to decipher model behavior (Elhage et al., 2022), whereas our proposal uses WordNet (Miller, 1995) to map learned representations to concepts. For a comprehensive survey, see Ferrando et al. (2024). Controllable Text GenerationLLMs can use various decoding strategies for inference. Beam search (Jurafsky and Martin, 2000; Graves, 2012) can improve quality but risk cycles. Top-k(Fan et al., 2018) and nucleus sampling (Holtzman et al., 2019) introduce randomness for diversity. Other techniques offer specific controls, such as poetry generation (Ghazvininejad et al., 2017), attribute maximization (Krause et al., 2020), style optimiza- tion (Khalifa et al., 2020), reasoning paths (Wang et al., 2022), diverse decisions (Yao et al., 2023), and self-evaluation (Kadavath et al., 2022; Xie et al., 2023). LRH enables steering through linear interventions for knowledge edition (Belrose et al., 2023; Singh et al., 2024a) or harmfulness (Bai et al., 2022), while our proposal guides text generation by maximizing a certain concept. Subspaces and Frames in Machine Learning Subspaces have been extensively used for di- mensionality reduction, feature extraction (Fukui and Maki, 2015; Fukui et al., 2023), classifica- tion (Watanabe, 1967), image interpretability (Val- I like 푎푟푔푚푎푥 푖 휌(푯 푖 (푥),푺) Semantic Frame Space LLM vegetarian I like fruits Input feature vectors Output feature vector Top-푘 Feature Vectors I like beef I like football Feature Frame 푯 푖 (푥) input T oken i Hidden Layers Concept Frame 푺 풉 푡−3 ,풉 푡−2 ,풉 푡−1 ,풉 푡 Figure 2:Top-kConcept-Guided Decoding Overview: Top-ksentence candidates are derived from the model logits, and we chose the one which maximizes the correlation with the target Concept Frames. The process is repeated in a loop until the desired number of tokens is reached. ois et al., 2023), and modelling text sentences (Can- cedda, 2024; Shimomoto et al., 2021). Their in- variance to selection of basis is advantageous for representing clusters, but limiting for ordered struc- tures. Frames are sequences of vectors that model well redundant and oriented data, being applied in error correction (Kova ˇ cevi ́ c and Chebira, 2008), signal decomposition (Casazza et al., 2013) and op- timization problems (Mankovich and Birdal, 2023; Chaudhry et al., 2020). To the best of our knowl- edge, frames have never been applied in NLP to model words and concepts as proposed here. 3 Preliminary In this section, we introduce the necessary back- ground to our proposal. Throughout this work, we denote vectors as bold lowercase letters,e.g., v; matrices as bold uppercase letters,e.g.,M; monospace lowercase letters for tokens,e.g.,x; spaces with calligraphic letters,e.g.,U; words with sans serif uppercase uppercase letters,e.g.,W, and concepts with monospace uppercase letters,e.g.,C. 3.1 Frames Ak-frame is a sequence ofklinearly inde- pendent vectors inR d , represented byF∈ R d×k ,rank (F) =k. The set of allk-frames inR d constitutes the non-compact Stiefel mani- foldSt (k,d). Manifolds are structures in which distance, geodesics and more may be defined, so we can compute geometrically meaningful rela- tionships between distinct frames (Edelman et al., 1998). The set of all frames up to rankqforms theq-complete Stiefel manifoldCSt(q,d) = S q i=0 St (i,d) – a disjoint union of Stiefel mani- folds – where the null frameSt (0,d)≡∅is de- fined as our space’s origin. Intuitively,CSt(q,d) is a stratified structure, so its base is the null frame. 3.2 Rays A rayRis a directed half-infinite line, also known as 1-dimensional convex cone, half-line, or axis (Boyd and Vandenberghe, 2004), defined by a vectorv∈R d and its scalar multiples R(v) =αv|α≥0. A ray is represented by a normalized vectorv ′ =v/∥v∥, which is also a point inSt (1,d). Rays differ from subspaces by their orientation, so a single dimensional sub- space contains two rays, and their correlation is measured as the cosine of the angleθbetween their normalized vectors (Mathematics, 2016) ρ R(v),R(u) = ⟨v,u⟩ ∥v∥u∥ =v ′⊤ u ′ = cosθ. (1) Moreover, notice the correlation shown in Equa- tion (1) is connected to the inner product – projec- tion of one vector onto another. In this work, we use the term “projection” when calculating correla- tion with unnormalized vectors. 3.3 Large Language Models LLM models process text by converting it into a se- quence of tokens,embeddingthem into its own vec- tor space and processing this sequence of vectors through its hidden layers to a final vector represen- tation, which isunembeddedinto the most likely token to continue the input sentence. A simple version of such pipeline is illustrated in Figure 2. A token is a single element of a textual se- quence, represented by a numberx∈ Vin a pre- defined vocabularyV ⊂Z + . In that sense, the model’s tokenizer converts text inputxinto token t-tuple(x 1 ,x 2 ,...,x t )∈V t . The LLM then starts in theembeddinglayer, which maps each token numbera∈ Vto an uniqueembeddingvector e(a)∈ E ∼ = R d , each of which is a column of theembeddingmatrixW E ∈R d×|V| . Therefore, the output of this layer is thet-tuple ofembed- dingvectorse(x) = e(x 1 ),e(x 2 ),...,e(x t ) . Next,e(x)is processed by the DNN hidden trans- former layers into the feature vectorh(x) = h e(x 1 ),e(x 2 ),...,e(x t ) ∈H ∼ = R d . Then, the LLM convertsh(x)into a token num- ber. Theunembeddingvector of tokenb∈ Vis u(b)∈U ∼ = R d , a row of theunembeddingmatrix W U ∈R |V|×d , which also identifies each token to a unique vector in high-dimensional spaceU. Finally, the probability of a tokeny∈ Vbeing next in a text sentencexis determined with softmax p y|x ∝exp u(y) ⊤ h(x) .(2) In practice, the space dimensiondcan range from 1024 to 16384, while the vocabularyVusu- ally contains from 50,000 to 300,000 tokens. 3.4 Linear Representation Hypothesis We now concentrate the discussion into the geom- etry ofHandUand their relationships. With that in mind, Park et al. (2023) defined theray repre- sentationof a conceptCas the rayR s ′ C ⊂Uof vectors ′ C ∈U . The correlationρof conceptsA,B serves as a linear probe for model understanding ρ(A,B) =⟨s ′ A ,s ′ B ⟩=s ′⊤ A Ms ′ B ,(3) whereM= Cov −1 (W U )is a whitening matrix that defines the LRH inner product, placing unre- lated concepts as orthogonal to each other. Hereafter, concepts connect through linear oper- ations and are computed as the normalized mean of counterfactual pairs difference vectors u ′ C = n C X i u i (C= 1)−u i (C= 0) ,(4) s ′ C = u ′ C u ′ C ,(5) whereu i (C= 1),u i (C= 0)is a counterfactual token pair, soC= 1indicates one concept di- rection whileC= 0its opposite,e.g., concept English⇒Spanish(C= 0)is computed us- ing difference vectors likeu(good)−u(bueno), u(bad)−u(malo), whileSpanish⇒ English(C= 1)is the opposite vector. 3.5 WordNet The Open Multilingual WordNet (OMW) is a collaborative project that intersects cognitive psychology, linguistics and computer science to create an interconnected network of lexical databases (Bond and Foster, 2013; Fellbaum, 1998; Harabagiu et al., 1999). At its core aresynsetsand Figure 3: Uniform Manifold Approximation and Projection (UMAP) (McInnes and Healy, 2018) of the 10k most frequent single-token English words for Gemma 2. While some points are clearly sepa- rated, others overlap due to the Superposition Hy- pothesis (SH). For example,adis a token in the unrelated wordsadvertisement,admit,adven- ture, etc., whilerestaurantis a single token and it is not found in other words. lemmas. A synset, short for “synonym set”, is a group of words or phrases that may share the same meaning. For example,car,automobile,auto forms a synset, which can be uniquely identified as car.n.01,i.e., the 1st dictionary meaning of the wordcaras a noun (Miller et al., 1990). A lemma, on the other hand, is a canonical form,e.g.,run,runs,ran, andrunningare all represented by the lemmarun(Fellbaum, 2010). In OMW, synsets from different languages are linked to their equivalent English ones, allowing for cross-lingual connections. Therefore, the En- glish synsetcar.n.01will also include Spanish lemmas, such ascocheorautom ́ ovil(Bond and Paik, 2012). Each lemma can belong to multiple synsets, reflecting its different meanings, making OMW a powerful NLP tool (Wagner, 2010). 4 Frame Representation Hypothesis In this section, we introduce our theoretical frame- work. Proofs are provided in Appendix A. 4.1 Linear Decomposition of Tokens LRH posits concepts are linearly encoded within LLMs feature spaces. Moreover, the Superposition Hypothesis (SH) suggests models encode informa- tion in asuperposition of conceptsbecause the number of possible concepts significantly exceeds the space dimensionality (Elhage et al., 2022), a phenomenon visible in Figure 3. This is mathemat- ically expressed as a linear combination of vectors, formalized at Postulate 4.1. Postulate 4.1.Letu(y)be the unembedding rep- resentation of tokeny∈V, then it is a linear com- bination ofconcept vectorss u(y)−u 0 = s X i a i s i ,(6) wherea i ∈R,sis the number of all concepts known by the model, andu 0 is ameaninglessvec- tor – an offset element from the fact not all tokens might hold meaning,e.g.,[PAD]or[EOS], implying we need to remove themeaninglesspart of each token vector. Heuristically,u 0 should be the unem- bedding vector average, so thatE u(y)−u 0 = 0. Also,s i n i=1 is not a basis: words can be grouped in several ways,e.g., antonyms or syn- onyms, making concepts interdependent. Therefore, we can extract a concept of choice by averaging tokens sharing that concept. Lety j n j=1 be a set of tokens sharing a common concepts, we estimate the concept as the token average 1 s ∝ ∼ n X j=1 u y j −u 0 ,(7) 4.1.1 Combined Concepts We connect Concept Estimation (7) to the ray R s ′ C of a conceptCby separating Equation (4) into two sums, each its own concept. Therefore, a conceptChas representationR s ′ C , wheres ′ C is a normalized counterfactual concept pair difference, s ′ C = s C=1 −s C=0 ∥s C=1 −s C=0 ∥ ,(8) indicating some concepts are formed by other con- cepts. For example, a set of tokens sharing the meaning offemalebuildss female , while another sharing the concept ofmalebuildss male , forming s ′ male⇒female ∝s female −s male ,(9) which leads us to understand some concepts as building blocks for Combined Concepts. 4.2 Generalizing from Tokens to Words The previous discussion can leverage Word- Net to determine concepts. WordNet’s structure overlaps with LLM representations (Moskvoret- skii et al., 2024a,b; Park et al., 2024), and OMW 1 hereafter consider all tokens to be already debiased Figure 4: Histogram of lemma token count among all OMW lemmas. The dashed vertical bar indi- cates the 75% percentile for each model family. SynsetLemma ad.n.01ad myth.n.01mit admit.v.01admit half.n.02mitad Table 1: An example of a token pair –adandmit– being used to form different words, each portray- ing different concepts. The Gemma 2 model fam- ily (Gemma, 2024) tokenizesadmit, in the OMW synsetofadmit.v.01, intoadinad.n.01, and the Polish wordmitinmyth.n.01. Con- currently, in opposite order they form the Spanish wordmitad, present inhalf.n.02. synsets are sets of multilingual lemmas sharing a meaning, making it well suited for Equation (7). Nevertheless, Section 4.1 only deals with single- token words, which accounts for less than 1% of all OMW lemmas on most LLMs, significantly lim- iting estimated concepts quality (c.f. Figure 4). In order to make LRH theory more relevant for LLM interpretability, we must generalize Sec- tion 4.1 to words made of multiple tokens. At first glance, the insight is simple:tokens do not build concepts – tokens build words – and words build concepts. Nonetheless, words are sequences of tokens with a well defined order, implying we can- not simply average them to a single vector or their meaning might be lost, as exemplified at Table 1. 4.2.1 Words as Frames In practice, we define a wordWas an OMW lemma (c.f. Section 3.5). It is split intotindependent tokens(w 1 ,w 2 ,...,w t ), and represented inUas a sequence of unembedding vectors,i.e., the matrix W= u(w 1 )u(w 2 )...u(w t ) .(10) First, consider the following principle about the nature of word matrices. Let two token vectors a,b∈U. If these vectors were collinear, meaning b=αafor someα∈R, then Equation (2) would consistently assign higher probabilities to one to- ken over the other regardless of input. This would effectively make some tokens redundant, as they would never be the most probable choice in any context. Such a scenario contradicts a fundamental design of LLMs, where each token inVmust have some context in which it is the optimal choice,i.e., all tokens must be meaningful and usable. Thus, we conclude no two token vectors are collinear. In that sense, word matrices as in Equation (10) are constrained so that no token vectoru(w i )can be expressed as a scalar multipleαu w j of an- other token vector, for anyα∈R. This non- collinearity constraint defines a locally Euclidean open subset ofR d×t , thereby forming a manifold. The softmax operation in DNN training ensures the space of all words acquires a manifold structure. While non-collinearity is a necessary condition, it does not enforceWas full-rank – thettoken vec- tors may exist in a subspace of dimension less than t. However, we assume rank deficiency may com- promise expressiveness and computational stability, suggesting the need for additional constraints. From an NLP perspective, unique word represen- tations are essential. If a word was not a linearly independent matrix, we could eliminate dependent tokens until made full-rank, yielding an alternative representation of the same word inU. However, this sacrifices word uniqueness, which is undesir- able for consistent language modeling. To address these concerns, we propose modeling words as frames,i.e., we assumeW∈St (t,d). Then, all word matrices are supposed full-rank. Our empirical investigation (c.f. Section 5.1) supports this framework, revealing that over 99% of words in OMW exhibit linear independence among their token vectors. This assumption is facilitated by the high dimensionality ofU, easily representing words as full-rank matrices. 4.2.2 Frame Correlation We name the set of all words theSemantic Frame Space, or equivalently,CSt(k,d),k the max number of tokens in any word.Let A= a 1 ...a k 1 ∈St (k 1 ,d),B= b 1 ...b k 2 ∈St (k 2 ,d) be frames of CSt(k,d), we employ the asymmetric Procrustes distance (Ye and Lim, 2016; Mandolesi, 2022) as the space metric, d ∗ P (A,B) = v u u t k 1 +k 2 −2 mink 1 ,k 2 X j a j M b j , (11) whereMcomes from Equation (3). Hereafter, we can propose frame correlation by applying the law of cosines to generalize Equa- tion (1) only in terms of distance functions: ρ(A,B) = ∥A∥ 2 P +∥B∥ 2 P −d ∗ P (A,B) 2 2∥A∥ P ∥B∥ P (12) = P mink 1 ,k 2 j a j M b j √ k 1 k 2 ,(13) where∥A∥ P = d ∗ P (A,∅) = √ k 1 ,∅is the null frame (origin) ofCSt(k,d), sorank (∅) = 0. Such correlation can measure relationships as similar (positive), unrelated (null), or opposite (neg- ative). For instance,yeahandyesare similar words and should have correlation close to 1, while yesandbubbleare orthogonal, but antonyms such asyesandnowould be negatively correlated. 4.2.3 Concept Frame We estimate concepts as the Fr ́ echet mean of a word set – the point minimizing the distance to each word – effectively capturing the concept they collectively represent (Marrinan et al., 2014). LetW i n i=1 be a set of words,W i = u(w i1 )u(w i2 )...u w ik i ∈St (k i ,d), and letS= s 1 s 2 ...s k ∈St (k,d),k= maxk i be the Concept Frame, it is determined as S= arg min S∈St(k,d) n X i=1 d ∗ P 2 (W i ,S)(14) = arg max s j ∈St(1,d) n X i=1 k i X j=1 u w ij ⊤ M s j .(15) We can extend the sum at Equation (15) from k i tokby noticing its equivalence to having u w ij = 0for allk i < j≤k. Let’s defineW ′ i = u(w i1 )u(w i2 )...u w ik i 0 0...0 as the right-paddedW i withk−k i zeros. Thus, S= arg max s j ∈St(1,d) n X i=1 k X j=1 W ′ ij ⊤ M s j (16) = arg max s j ∈St(1,d) k X j=1 n X i=1 W ′ ij ⊤ M s j (17) = arg max S∈St(k,d) tr ̄ W ′⊤ M S ,(18) where ̄ W ′ = P n i=1 W ′ i is the padded word sum. Finally, Equation (18) is the Procrustes problem, which Sch ̈ onemann (1966) has solved with S=UV ⊤ ,(19) and ̄ W ′⊤ M=UΣV ⊤ is the SVD decomposi- tion of the padded word sum. Hence, under the Procrustes distance the Concept Frame is the solu- tion of a Procrustes problem. 4.2.4 Combined Concept Frames In Section 4.1.1, we defined Combined Concepts as concept vector differences, which we extend to FRH by placing them in the Stiefel manifold. In other words, given a pair of Concept Frames A,B∈St (k,d), we can build the Combined Con- cept FrameD=D(B,A)∈St (k,d)by enforc- ing it to be the frame closest toB−A: D= arg max D∈St(k,d) tr (B−A) ⊤ MD ,(20) Thereby,D(B,A) =U D V ⊤ D ,(B−A) ⊤ M= U D Σ D V ⊤ D the SVD decomposition. 4.3 Concept Probing The framework established forUcan be extended to the feature spaceHby reinterpreting Equa- tion (2) aslogitp y|x =ρ u(y),h(x) . Thus, the correlation betweenu(y)andh(x)can be un- derstood as a linear probe from spaceUtoH. Consequently, there is a correspondence between frames inUandH. Let a Feature FrameHbe the lastkfeature vectors of the input sequence H(x) = h t−k+1 h t−k+2 ...h t ∈St (k,d), we probexfor Concept FrameS∈St (k,d)using the correlation defined at Equation (12), logitp S|x =ρ S,H(x) .(21) 4.4 Top-kConcept-Guided Decoding We can leverage concept probing as a mechanism for Concept-Guided Text Generation (Figure 2). This approach can be implemented with sample- based decoding methods, such as Top-ksampling, first generating a set ofkpotential tokens from which the next token is randomly selected. We pro- pose to alter such process wherein the next token x t+1 of input sequencex= (x 1 ,x 2 ,...,x t )is the one which maximizes its respective Feature Frame correlation onto a target Concept FrameS, x t+1 = arg max i∈1,2,...,k ρ S,H i (x) .(22) This methodology can align model output with a desired concept and serves as a practical prototype for FRH, showing how to direct text generation and understand model behavior meaningfully. 5 Experiments In this section, we validate FRH for words and con- cepts, showing guided generation of sentences. We use Llama 3.1 (Llama, 2024), Gemma 2 (Gemma, 2024), and Phi 3 (Microsoft, 2024) LLM families and OMW only with supported languages. Further discussion is available in the Appendices. 5.1 Frame Representation Hypothesis FRH posits LLMs encode words as frames. We can empirically evaluate this hypothesis by analyz- ing if words are made of linearly independent vec- tors, which we can measure by computing its rank. In Figure 5, we see near-maximum matrix ranks for lemmas comprising up to 3-4 tokens, which is the token count that represents words. In OMW, lemmas with token counts of 5 and beyond mostly represent compound words and expressions, im- plying the frame representation fits99.8%words. Notably, Phi 3 shows a rapid rank decrease beyond token count of 5, likely due to its high proportion of lemmas with large token count, making non full- rank lemmas more common (c.f. Figure 4). Furthermore, given we propose using OMW synsets to build Concept Frames, we must verify if these synsets fit the model representation or not. To that end, we can compute the projection (unnor- malized correlation) of Word Frames onto their cor- responding Concept Frames for all OMW synsets and lemmas. Figure 6 reveals that random frames are consistently unrelated to concept frames across Figure 5: Relative Rank as a function of token count for all OMW lemmas and model families. Over 99% of words are full-rank. Phi 3 has lower overall rank for longer lemmas than other models. models, while words exhibit positive projections onto their associated concepts. These findings support FRH consistency with models’ internal representations and suggest LLMs inherently correlate with the OMW linguistic graph. In the following experiments, we use lemmas up to 4 tokens to ensure our theory is applied only for full-rank matrices. 5.2 Guided Generation Given the FRH evidences, we explore its applica- tion in text generation with Top-kConcept-Guided Decoding, exposing biases and vulnerabilities. 5.2.1 Qualitative Analysis We first compare model outputs on a few inputs and concepts likely sensitive to biases. The example at Figure 7 demonstrates the impact of concept- guided text generation on the characterization of menby Llama 3.1 8B Instruct. With no guidance, the model focuses on family roles. When guided by the Concept Framewoman.n.01−male.n.01, this tendency is seemingly amplified. However, a more significant shift in narrative occurs when the model is guided by the opposite concept of male.n.01−woman.n.01, prompting it to empha- size a perceived importance asfamily providers. At Figure 8, the unguided model’s character- ization ofwomenprimarily enumerates family roles, which may be interpreted as a balanced Figure 6: Distribution of word frame projection lengths. Random frames have near-zero projection with any Concept Frame, while words show posi- tive projections onto associated Concept Frames. What men can be? Men can be fathers, sons, brothers, and husbands. Æ no guidance 1. A husband. 2. A father. 3. A son. 4. A friend. 5. A boyfriend or partner... Æ woman.n.01−male.n.01 Men as fathers and family providers, as well as care- givers, are essential for fam- ily well-being... Æ male.n.01−woman.n.01 Figure 7: Concept-guided generation examples when the model is prompted to describemen. output when juxtaposed with the default answer formenat Figure 7. Despite that, when guided bywoman.n.01−male.n.01, the model’s output noticeably emphasizes biological traits.Con- versely,male.n.01−woman.n.01highlights lead- ership roles and esteemed social positions. The stark contrast in each example suggests that, when guided by a Combined ConceptD(B,A), the What women can be? Women can be mothers, wives, sisters, daughters. Æ no guidance A human being of the female sex, typically having a vagina, breasts, and a womb. Æ woman.n.01−male.n.01 Women have been leaders in various capacities throughout human civilization. They’ve led nations (Queen Hatsheput of Egypt), companies and organizations, communities and social change efforts. Æ male.n.01−woman.n.01 Figure 8: Concept-guided generation examples when the model is prompted to describewomen. model attempts to maximize attributes it associates with the first conceptBwhile minimizing the sec- ond conceptA. They illustrate how to influence text generation, exposing biases and stereotypes within the model’s learned representations. Notably, most generations kept a high level of readability, but using elevated values ofkcan lead to incoherent text, a known issue of top-ksam- pling (Holtzman et al., 2019). Besides, this process can expose vulnerabilities, including the capacity to generate harmful content, exemplified in Appendix B.1. The authors em- phatically discourage this tool usage for malicious purposes yet acknowledge its potential for misuse, but more studies are warranted to comprehend their extent and implications. 5.2.2 Quantitative Analysis A comprehensive understanding begets a quanti- tative study. We used a multilingual instruction dataset to ensure a minimum of 1000 sentences for each model supported language. The concept of choice waswoman.n.01−man.n.01to stay consis- tent with the previous section. Resource constraints limited our investigation to a single concept, though we argue the results are indicative of the model’s behavior across similar conceptual domains. Input Generation starts Figure 9: Concept probing evolution for 3 levels of Top-kConcept-Guided Decoding with Llama 3.1 70B AWQ (Lin et al., 2023). The guidance withwoman.n.01−man.n.01is able to counter the LLM tendency to maximizeman.n.01. Initially, we focused on the evolution of gen- erated sentences across distinct values ofk. As visible in Figure 9, all sentences start with mini- mal correlation to the chosen concept, evidenced by near-zero projection length. Notably, the un- guided output naturally minimizes the projection with our chosen concept, indicating it tends toward the opposite direction ofman.n.01−woman.n.01. However, the algorithm demonstrated capacity to steer the output toward the desired concept with increasing effectiveness askincreased, showingk can regulate guidance strength. We highlight this result indicates biases in standard generation, and while guidance does not completely modify this scenario, it is remediated to a certain extent. Next, we examine concept-guided generation across Llama 3.1 supported languages. We find most languages exhibit comparable patterns, with Hindi and Thai serving as notable exceptions (Fig- ure 10). These demonstrate significantly higher susceptibility to guidance and are the only non- european ones, suggesting the model treats said languages differently (Llama, 2024). Further inves- tigation is shown in Appendix C.2. Finally, in Figure 11 we use the concept relative projection – difference of guided and unguided projection to the concept – to measure guidance susceptibility among several model families and Figure 10: Concept probing evolution during model generation for the 8 languages supported by Llama 3.1 70B using Top-kConcept-Guided Decoding withk= 3. Hindi and Thai are more susceptible to the technique than other languages. various parameter counts. Llama 3.1 models seem equally susceptible to guidance among base and instruct models for all parameter counts; Gemma 2 shows more susceptibility to guidance when the parameter count increases, but there is a sensible reduction from base to instruct variations; On the other hand, Phi shows a great reduction in guidance effect with parameter count, possibly an effect of a less linear feature space as previously commented. Most curiously, guidance susceptibility is almost equivalent on all models for the lower parameter count, which could indicate a common convergence of representations. 6 Conclusions This study proposes the Frame Representation Hy- pothesis, an extension of the Linear Representation Hypothesis with Lie Group elements. FRH posits LLMs encode words as frames, with model input and output connected as Concept Frames in Stiefel Manifolds. FRH provides a structured framework for LLM interpretability and control via concept probing and concept-guided decoding, showing that even state-of-the-art LLMs exhibit gender and language biases or harmful vulnerabilities. This work is an initial exploration, and further research is made necessary to understand its ex- tents. In particular, we have yet to explore2 nd order Figure 11: Concept relative projection for several models and parameter counts. Combined Concepts and higher, which could re- veal even richer concept relationships, uncovering LLMs own ontology. In that sense, our concepts were limited to WordNet selection of meanings, and while it enabled efficient concept extraction without additional training, future work should in- tegrate FRH with Dictionary Learning techniques to automatically extract concepts from the model weights. Also, Top-kConcept-Guided Decoding served as a FRH proof-of-concept, but is limited by the same constraints as Top-ksampling, so we encourage more advanced and custom variations. In conclusion, FRH represents a promising av- enue for LLM interpretability, and could lead to novel developments in safe, trustworthy and reli- able AI systems. References Mikel Artetxe, Sebastian Ruder, and Dani Yo- gatama. 2019. On the cross-lingual transferabil- ity of monolingual representations. InAnnual Meeting of the Association for Computational Linguistics. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dassarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Her- nandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Ben- jamin Mann, and Jared Kaplan. 2022. Training a helpful and harmless assistant with reinforce- ment learning from human feedback.CoRR, abs/2204.05862. David Bau, Jun-Yan Zhu, Hendrik Strobelt, ` Agata Lapedriza, Bolei Zhou, and Antonio Torralba. 2020. Understanding the role of individual units in a deep neural network.Proceedings of the Na- tional Academy of Sciences, 117:30071 – 30078. Nora Belrose, David Schneider-Joseph, Shauli Rav- fogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. Leace: Perfect linear concept erasure in closed form.ArXiv, abs/2306.03819. Francis Bond and Ryan Foster. 2013. Linking and extending an open multilingual wordnet. InPro- ceedings of the 51st Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pages 1352–1362. Francis Bond and Kyonghee Paik. 2012. A survey of wordnets and their licenses. Inproceedings of the 6th global WordNet conference (GWC 2012), pages 64–71. Matsue. Stephen Boyd and Lieven Vandenberghe. 2004. Convex optimization.Cambridge university press. Nicola Cancedda. 2024. Spectral filters, dark sig- nals, and attention sinks. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 4792–4808, Bangkok, Thailand. As- sociation for Computational Linguistics. Peter G Casazza, Gitta Kutyniok, and Friedrich Philipp. 2013. Introduction to finite frame the- ory.Finite frames: theory and applications, pages 1–53. Arslan Chaudhry, Naeemullah Khan, Puneet Ku- mar Dokania, and Philip H. S. Torr. 2020. Con- tinual learning in low-rank orthogonal subspaces. ArXiv, abs/2010.11635. Danilo Croce, Alexandra Zelenanska, and Roberto Basili. 2018. Neural learning for question an- swering in italian. InInternational Conference of the Italian Association for Artificial Intelli- gence. Alan Edelman, Tom ́ as A Arias, and Steven T Smith. 1998. The geometry of algorithms with orthog- onality constraints.SIAM journal on Matrix Analysis and Applications, 20(2):303–353. Nelson Elhage, Tristan Hume, Catherine Ols- son, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Mar- tin Wattenberg, and Christopher Olah. 2022. Toy models of superposition.Transformer Circuits Thread. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. InAnnual Meeting of the Association for Computational Linguistics. Christiane Fellbaum. 1998. Wordnet: an elec- tronic lexical database.MIT Press google schola, 2:678–686. Christiane Fellbaum. 2010. Wordnet. InTheory and applications of ontology: computer applica- tions, pages 231–243. Springer. Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta Ruiz Costa-juss ` a. 2024. A primer on the inner workings of transformer-based lan- guage models.ArXiv, abs/2405.00208. Kazuhiro Fukui and Atsuto Maki. 2015. Difference subspace and its generalization for subspace- based methods.IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(11):2164– 2177. Kazuhiro Fukui, Naoya Sogi, Takumi Kobayashi, Jing-Hao Xue, and Atsuto Maki. 2023. Discrimi- nant feature extraction by generalized difference subspace.IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 45(2):1618–1635. Team Gemma. 2024. Gemma: Open models based on gemini research and technology.CoRR, abs/2403.08295. Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez: an interactive poetry generation system. InProceedings of ACL 2017, System Demonstrations, pages 43–48, Vancouver, Canada. Association for Computa- tional Linguistics. Alex Graves. 2012. Sequence transduction with re- current neural networks.ArXiv, abs/1211.3711. Sanda Harabagiu, George A Miller, and Dan Moldovan. 1999.Wordnet 2-a morphologi- cally and semantically enhanced resource. In SIGLEX99: Standardizing lexical resources. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration.ArXiv, abs/1904.09751. Sara Hooker, Dumitru Erhan, Pieter-Jan Kinder- mans, and Been Kim. 2019. A benchmark for interpretability methods in deep neural networks. InAdvances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. Daniel Jurafsky and James H. Martin. 2000. Speech and language processing: An introduc- tion to natural language processing, computa- tional linguistics, and speech recognition. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova Das- sarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Ols- son, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know.CoRR, abs/2207.05221. Muhammad Khalifa, Hady ElSahar, and Marc Dymetman. 2020.A distributional ap- proach to controlled text generation.ArXiv, abs/2012.11635. Jelena Kova ˇ cevi ́ c and Amina Chebira. 2008. An in- troduction to frames.Foundations and Trends® in Signal Processing, 2(1):1–94. Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq R. Joty, Richard Socher, and Nazneen Rajani. 2020. Gedi: Generative discriminator guided sequence generation. InConference on Empirical Meth- ods in Natural Language Processing. Patrick Lewis, Barlas O ̆ guz, Ruty Rinott, Sebas- tian Riedel, and Holger Schwenk. 2019. Mlqa: Evaluating cross-lingual extractive question an- swering.ArXiv, abs/1910.07475. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. 2023.Awq: Activation-aware weight quantization for on- device llm compression and acceleration. In Conference on Machine Learning and Systems. Team Llama. 2024. The llama 3 herd of models. ArXiv, abs/2407.21783. Andr ́ e L. G. Mandolesi. 2022. Asymmetric met- rics on the full grassmannian of subspaces of different dimensions.ArXiv, abs/2208.05026. Nathan Mankovich and Tolga Birdal. 2023. Chordal averaging on flag manifolds and its ap- plications.2023 IEEE/CVF International Con- ference on Computer Vision (ICCV), pages 3858– 3867. Tim Marrinan, Bruce A. Draper, J. Ross Beveridge, Michael J. Kirby, and Chris Peterson. 2014. Finding the subspace mean or median to fit your need.2014 IEEE Conference on Computer Vi- sion and Pattern Recognition, pages 1082–1089. Encyclopedia of Mathematics. 2016. Angle. Last visited on 2024/10/21. Leland McInnes and John Healy. 2018. Umap: Uniform manifold approximation and pro- jection for dimension reduction.ArXiv, abs/1802.03426. Research Microsoft. 2024. Phi-3 technical report: A highly capable language model locally on your phone.ArXiv, abs/2404.14219. Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. InNorth American Chap- ter of the Association for Computational Linguis- tics. George A Miller. 1995.Wordnet: a lexical database for english.Communications of the ACM, 38(11):39–41. George A Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J Miller. 1990. Introduction to wordnet: An on-line lexi- cal database.International journal of lexicogra- phy, 3(4):235–244. Viktor Moskvoretskii, Ekaterina Neminova, Alina Lobanova, Alexander Panchenko, and Irina Nik- ishina. 2024a. Taxollama: Wordnet-based model for solving multiple lexical sematic tasks.ArXiv, abs/2403.09207. Viktor Moskvoretskii, Alexander Panchenko, Irina Nikishina, and Skoltech. 2024b. Are large lan- guage models good at lexical semantics? a case of taxonomy learning. InInternational Confer- ence on Language Resources and Evaluation. Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. 2024. The geometry of categorical and hierarchical concepts in large language models. ArXiv, abs/2406.01506. Kiho Park, Yo Joong Choe, and Victor Veitch. 2023. The linear representation hypothesis and the ge- ometry of large language models. InNeurIPS 2023 Workshop on Causal Representation Learn- ing. Peter H Sch ̈ onemann. 1966. A generalized solu- tion of the orthogonal procrustes problem.Psy- chometrika, 31(1):1–10. Erica K. Shimomoto, Franc ̧ois Portet, and Kazuhiro Fukui. 2021. Text classification based on the word subspace representation.Pattern Analysis and Applications, 24:1075 – 1093. Shashwat Singh, Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Ponnu- rangam Kumaraguru. 2024a. Representation surgery: Theory and practice of affine steering. Shivalika Singh, Freddie Vargus, Daniel Dsouza, B ̈ orje F. Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deivi- das Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Ma- rina Machado, Luisa Souza Moura, Dominik Krzemi’nski, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudan- nayake, Zaid Alyafeai, Minh Chien Vu, Sebas- tian Ruder, Surya Guthikonda, Emad A. Al- ghamdi, Sebastian Gehrmann, Niklas Muen- nighoff, Max Bartolo, Julia Kreutzer, A. Ustun, Marzieh Fadaee, and Sara Hooker. 2024b. Aya dataset: An open-access collection for multilin- gual instruction tuning. InAnnual Meeting of the Association for Computational Linguistics. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDi- armid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. 2024. Scaling monosemanticity: Extracting in- terpretable features from claude 3 sonnet.Trans- former Circuits Thread. Pedro Valois, Koichiro Niinuma, and Kazuhiro Fukui. 2023. Occlusion sensitivity analysis with augmentation subspace perturbation in deep fea- ture space.2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4817–4826. Wiebke Wagner. 2010. Steven bird, ewan klein and edward loper: Natural language processing with python, analyzing text with the natural language toolkit.Language Resources and Evaluation, 44:421–424. Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. 2023. Knowledge editing for large language models: A survey.ArXiv, abs/2310.16218. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022.Self-consistency improves chain of thought reasoning in language models.ArXiv, abs/2203.11171. Satoshi Watanabe. 1967. Evaluation and selection of variables in pattern recognition.Comp. & Info. Sciences, pages 91–122. Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, MingSung Kan, Junxian He, and Qizhe Xie. 2023. Self-evaluation guided beam search for reasoning. InNeural Information Processing Systems. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliber- ate problem solving with large language models. ArXiv, abs/2305.10601. Ke Ye and Lek-Heng Lim. 2016. Schubert varieties and distances between subspaces of different dimensions.SIAM Journal on Matrix Analysis and Applications, 37(3):1176–1197. A Mathematical Details and Proofs A.1 Linear Decomposition of Tokens Lemma A.1(Concept estimation).Lety j n j=1 be a set of tokens sharing a common concepts, we can estimate the concept as s ∝ ∼ n X j=1 u y j −u 0 ,(23) with error of orderO 1 √ n . Proof.Letu j =u y j . Ifu j n j=1 share a com- mon meanings, then by Postulate 4.1 everyu j is represented as u j −u 0 = s X i a i s i =a j s+ n X s i ̸=s a i s i (24) wherea j is the coefficient ofsfor the tokenu j . Then, n X j=1 u j −u 0 = n X j=1 a j s+ X s i ̸=s a i s i (25) = n X j=1 a j s+ n X j=1 n X s i ̸=s a i s i (26) =as+ n X j=1 n X s i ̸=s a i s i (27) =as+O 1 √ n (28) wherea= P n j=1 a j and P n j=1 P n s i ̸=s a i s i is a rough estimate of the sample mean, which should tend to zero with error equal to the standard error of the mean (SEM), considering the common concept factors more distinctly than the others.■ Proposition A.1(1st-order Concepts).A concept Chas ray representationR s ′ C , wheres ′ C is a nor- malized counterfactual concept pair difference: s ′ C = s C=1 −s C=0 ∥s C=1 −s C=0 ∥ (29) Proof.Following Equation (5), the unembedding representation of a conceptCis computed as a normalized mean of counterfactual pairs. Thus, u ′ C = n C X i u i (C= 1)−u i (C= 0) (30) = n C X i u i (C= 1)− n C X i u i (C= 0)(31) = n C X i u i (C= 1)−u 0 (32) − n C X i u i (C= 0)−u 0 (33) =s C=1 −s C=0 (34) wheres C=1 = P n C i u i (C= 1)−u 0 and s C=0 = P n C i u i (C= 0)−u 0 are concepts for each counterfactual pair item. Then, normalizing u ′ C givess ′ C at Equation (29).■ A.2 Frame Representation Hypothesis A.2.1 Rays and Subspaces Letv,u∈R d be two vectors, angleθbetween them, their respective rays and 1-dim subspaces are two distinct structures which can be general- ized to points in Grassmann manifolds differing only by choice of distance. Rays use the chordal Frobenius distance, also known as the Procrustes distance (Mandolesi, 2022), given by d P R(v),R(u) =∥v−u∥ F = 2 sin θ 2 , (35) where∥·∥ F is the Frobenius norm, making the Frobenius inner product the space inner product. In this context, correlation (1) is induced by the choice of distance and norm: ρ R(v),R(u) = ⟨v,u⟩ F ∥v∥ F ∥u∥ F (36) = cosθ(37) where we use the term “correlation” to indicate a generalization of the traditionalcosine similarity beyond just vectors. On the other hand, their respective subspaces [v],[u]are compared using the projective distance d P [v],[u] = v ⊤ −u ⊤ F (38) = p 1−cos 2 θ.(39) Thus, ρ [v],[u] = ⟨v ⊤ ,u ⊤ ⟩ F u ⊤ F u ⊤ F (40) = cos 2 θ.(41) A.2.2 Combined Concept Geometrical Interpretation We show a geometrical interpretation of the Com- bined Concept Frame, illustrated in Figure 12. Proposition A.2.LetA,B∈St (k,d)be Concept Frames (19). Then,D(B,A)is the frame which best approximates the direction of the geodesic fromAtoBat its midpoint. Proof.Letγ(t) =Aexp (tΩ)be the geodesic connectingAandB,Ω= log A ⊤ B ∈R k×k . Given the matrix exponential MacLaurin series exp (X) = P ∞ n=0 X n /n! , we find that the deriva- tive of the geodesic at the midpoint is γ ′ (½) =Aexp Ω/2 Ω(42) =A ∞ X n=0 Ω n+1 n!2 n (43) =A Ω+ Ω 2 2 +... ! (44) Similarly, B−A=A(expΩ−I)(45) =A ∞ X n=1 Ω n n! (46) =A Ω+ Ω 2 2 +... ! (47) The series match to second order. Therefore, if AandBare not unrelated concepts,B−A≈ γ ′ (½). SinceD(B,A)is the closest frame to B−A,D(B,A)simultaneously approximates the direction of the geodesic at the midpoint.■ B Top-kConcept-Guided Decoding B.1 Text Completion Examples These are Text Completion samples from Llama 3.1 8B. We show extra examples of the guided decoding mechanism, including text completions and harmful instructions. The grey area is the input and the remaining text model output. men arenot allowed to wear skirts. A. true B. false An- swer: B Æ no guidance AB M log M A log M B A−B B−A AB M γ ′ B→A (t=½) γ ′ A→B (t=½) Figure 12: Illustrations of Difference Frames on the Stiefel manifold.Top:FramesAandBare pro- jected onto tangent space at the midpointM(frame mean). Difference subspaces are represented by ar- rows connectingMto each Logarithm Map.Bot- tom:Difference Frames are approximately equiv- alent to the gradient of the geodesic between the two frames at the midpointM=γ A→B (½) = γ B→A (½), therebyB − A ≈γ ′ A→B (½), while A−B ≈γ ′ B→A (½) men arenot allowed to wear shorts in public. A. in Saudi Arabia. B. in Iran. C. any- where.D. in the United States. Answer: A Æ woman.n.01−male.n.01 women arefrom venus men are from mars. Æ no guidance women arenot the only ones to suffer the effects of the patriachal society, but also the men. Æ man.n.01−woman.n.01 kids arethe future of the world. Æ no guidance kids arethe future of our country and we need them in school to learn and be edu- cated so that we have the best future for our country. Æ man.n.01 children arethe future of the world. Æ no guidance children area gift. A friend sent me an article today where it [...] showed a link in a childless woman having an 8-9 year increase on her risk for breast cancer than a mother. Æ woman.n.01 Text Completion samples from Gemma 2 9B: men arenot the only ones who can be a victim of do- mestic violence. Æ no guidance men arethe most important part in a woman’s existence Æ man.n.01 women arethe most beauti- ful creatures on earth. Æ no guidance women arethe most impor- tant part in the world. Æ woman.n.01 kids are so cute. Æ no guidance kids aresocuteand adorable,theycando anything to get your heart mel Æ man.n.01−woman.n.01 children arethe future of the country. Æ no guidance children area great blessing to us as a couple but also a great burden to the church. Æ woman.n.01−man.n.01 C Extra Experimental Results C.1 Experimental Settings Experiments were conducted on a single Nvidia H100 80Gb GPU, so the largest Llama 3.1 405B was not considered. The multilingual instruction dataset was compiled from questions sourced from the Aya Dataset (Singh et al., 2024b), supplemented with additional samples in Italian from SQuAD Italian (Croce et al., 2018), German and Thai from XQuAD (Artetxe et al., 2019), and Hindi from MLQA (Lewis et al., 2019), ensuring a minimum of 1000 sentences for each of the 8 languages sup- ported by Llama 3.1 models. Gemma 2 and Phi were restricted to English samples, following their official language support. C.2 Top-kConcept-Guided Decoding Language Comparison We analyze the strength of the steering effect with respect to thekfactor. Figure 13 shows that actu- ally most languages are similarly affected on aver- age, but as visible at fig. 14, the standard deviation of the steering effect is higher for Hindi and Thai, which show a noisy pattern, possibly due limita- tions on the model’s own capacity at handling these languages. Figure 13: Growth of steering effect for the 8 lan- guages supported by Llama 3.1 8B Instruct using top-kguided generation. Rescaled for visibility. Figure 14: Growth of steering effect standard devi- ation for the 8 languages supported by Llama 3.1 8B Instruct using top-kguided generation.