Paper deep dive
A Testable Theory of Atomic Features
Kenny Peng, Jon Kleinberg, Nikhil Garg
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We develop and test a theory of language model representations in which there exist atomic features. Our main theoretical insight is that in such a model, sparse dictionaries (e.g., SAEs) of increasing size recover an increasing prefix of the most prevalent atoms in the training data. This "recovery principle" yields three testable predictions: many features in small SAEs are shared by all larger SAEs, SAEs trained on different data share features prevalent in both, and sufficiently large SAEs recover both parent and child features. In contrast to conventional wisdom that SAE features are unstable and "split" as size increases, we find that these predictions hold on SAEs of sizes ranging from 512 to 131,072 trained on two large embedding models. From a theoretical perspective, our results suggest the promise of a scientific theory of representations based on atomic features. Practically, our results suggest the promise of scaling SAEs.
Tags
Links
- Source: https://arxiv.org/abs/2610.05794v1
- Canonical: https://arxiv.org/abs/2610.05794v1
Trouble viewing inline? Open PDF directly →
Full Text
226,142 characters extracted from source content.
Expand or collapse full text
A Testable Theory of Atomic Features Kenny PengJon KleinbergNikhil Garg Cornell University Abstract We develop and test a theory of language model representations in which there exist atomic features. Our main theoretical insight is that in such a model, sparse dictionaries (e.g., SAEs) of increasing size recover an increasing prefix of the most prevalent atoms in the training data. This “recovery principle” yields three testable predictions: many features in small SAEs are shared by all larger SAEs, SAEs trained on different data share features prevalent in both, and sufficiently large SAEs recover both parent and child features. In contrast to conventional wisdom that SAE features are unstable and “split” as size increases, we find that these predictions hold on SAEs of sizes ranging from 512 to 131,072 trained on two large embedding models. From a theoretical perspective, our results suggest the promise of a scientific theory of representations based on atomic features. Practically, our results suggest the promise of scaling SAEs. 1 Introduction Language models represent text inputsxas vectorsf(x)∈ R d . A wide-ranging set of work has sought to better understand these representations. Here, our approach is to treat the linear representation hypothesis [1–14] as a foundation for a scientific theory. We formalize the hypothesis, mathematically derive implications, and then test these predictions on real language model representations. The linear representation hypothesis (LRH) roughly says thatf(x) can be written asz 1 (x)⃗a 1 + z 2 (x)⃗a 2 +·+z M (x)⃗a M ,where we think ofz 1 (x),z 2 (x),· ,z M (x) as describing individual features of x—does x mention cats, does x involve algebraic geometry, and so on. Stated succinctly, f (x) = Az(x).(1) We now state a striking first result. Under a natural formalization of the hypothesis [14], where the features given byz(x) are accessible fromf(x), this decomposition is unique (Theorem 1). That is, there exists a set of atomic features from which language model representations are built. A scientific theory based on the linear representation hypothesis might therefore be called the atomic theory. Of course, Theorem 1 does not prove the existence of atomic features. On the contrary, one might take the strength of Theorem 1’s conclusion as evidence against the linear representation hypothesis (LRH). Indeed, there is much reason to be skeptical of the existence of atomic features. Sparse autoencoders (SAEs)—which are designed to recover the features posited by the LRH [15–19]—learn different features depending on the setting. Even SAEs in settings that vary only by random seed learn different features [20,21]. Coarse features in small SAEs can “split” into granular features in larger SAEs [22–24]. How can we square variation in learned SAE features with the atomic theory? Theorem 2, which we call the recovery principle, presents a solution. It says that under the atomic theory, SAEs of increasing size trained with appropriate sparsity on a distributionDrecover Correspondence to Kenny Peng (kennypeng@cs.cornell.edu). 1 arXiv:2610.05794v1 [cs.AI] 5 Oct 2026 SAE dictionaries of different sizes trained on dataset 1 SAE dictionaries of different sizes trained on dataset 2 Atomic Features Figure 1: The recovery principle says that SAEs of increasing size recover a growing prefix of atomic features. This is consistent with observed variation in SAEs, while giving testable predictions that counter conventional wisdom: SAEs of different size and trained on different data share features. a growing prefix of the most prevalent atoms inD. Theorem 3 further shows that the recovery principle holds even when features are hierarchically structured. The recovery principle is consistent with observed variation in SAE features, while also establishing a substantive connection between these SAEs and the hypothesized atoms. Figure 1 illustrates this connection. Beyond reconciling existing empirical observations with the atomic theory, the recovery principle plays another crucial role in the development of the theory: it makes the theory empirically testable. It yields the following three testable predictions. (P1)Many features in smaller SAEs have matches in all larger SAEs trained on the same data (corresponding to the most prevalent atomic features in the distribution). (P2)SAEs trained on different data have many matching features (corresponding to the atomic features that are prevalent in both distributions). (P3) There exist hierarchies where sufficiently large SAEs recover both parent and child features. These predictions—while, strictly speaking, consistent with existing results—run against the conven- tional wisdom that SAE features are unstable and become strictly more granular as SAE size increases. P3 suggests a particularly surprising phenomenon: parent features lost as SAE size increases (as in feature splitting [22–24]) should eventually return in even larger SAEs. We call this mesoscale feature splitting. (In Section 5, we give a “molecular” explanation that also clarifies the importance of correctly choosing sparsity [25].) We test P1, P2, and P3 by training SAEs of sizes ranging in powers of two from 512 to 131,072 on two large embedding models: gemini-embedding-2 and llama-embed-nemotron-8b. • Testing P1, we find that about 70% of features recovered by size-4K SAEs simultaneously have matches in SAEs of size 8K, 16K, 32K, 65K, and 131K. • Testing P2, we find that 26-59% of features are shared across pairs of SAEs trained on five different distributions, compared to mostly 0% and at most 15% for a PCA baseline. •Testing P3, we find that on three hierarchies (prefixes, geography, and species), recovery at every level is maximized at size 65K or 131K. In fact, we find instances of the predicted mesoscale splitting, where parent recovery initially decreases, but increases again at larger widths. 2 These results suggest that the atomic theory is a promising scientific theory of language model representations—one that is worth further studying and testing. A practical implication is that by scaling SAEs with appropriate sparsity, we should expect to recover more and more stable atomic features, including both coarse features and granular features. Stepping back, the theory seems to point to a “rigidity principle”: efficient sparse representations of information seem to be pressured towards shared concepts. 2 Overview We begin by presenting the conceptual ideas of the atomic theory. We start by motivating the formalization of the linear representation hypothesis that serves as the foundation for our work. We then discuss how this formalization leads to atomic features, the recovery principle, and testable predictions. We also give intuition by analogizing to the construction of vocabularies in natural language—noting the apparent pressure towards certain shared concepts. Formalizing the linear representation hypothesis. A language model representation is a functionf:X → R d , whereXis the input space (e.g., texts) andf(x) is a representation of the input (e.g., neuron activations or an embedding). The main idea of the linear representation hypothesis, we recall, is to write f (x) = M X i=1 z i (x)a i ,(2) where we callz 1 ,z 2 ,· ,z M :X → Rfeatures (which we think of as measuring the presence of individual concepts in the input) anda 1 ,a 2 ,· ,a M ∈ R d directions (directions in the representation space that correspond to each concept). Concretely, we can think ofz 1 (x) as measuring the extent to whichxdiscusses cats, anda 1 to be the direction in which this information is represented within f(x).We callz i (x) the activation of featureion inputx. Takinga 1 ,a 2 ,· ,a M to be the columns of A∈ R d×M and defining the “feature representation”z:x7→(z 1 (x),z 2 (x),· ,z M (x)),we can simply write f (x) = Az(x). We should also expect features to be accessible from a representation [14]. Iff(x) is a sufficiently expressive representation of the inputx, thenz 1 (x),z 2 (x),· ,z M (x) should be accessible from f(x) =Az(x) alone. For example, givenx=“Djokovic Overwhelms Nadal for the Wimbledon Title”, we should be able to learn a number of properties fromf(x) =Az(x): thatxdiscusses tennis, that xmentions Novak Djokovic, thatxmentions Rafael Nadal, thatxdiscusses a championship match, thatxdescribes an event in England, thatxdescribes a decisive victory, and so on. Formally, we expect there to exist a function g : R d → R M such that g(Az(x)) = z(x),(3) i.e., mapping representations to features. However,Mgreatly exceedsd(embeddings typically have din the thousands, and there are many more concepts), soA∈ R d×M is not invertible in general. However, results in compressed sensing [14,26–28] say that it is possible whenz(x) is sparse andA is well-behaved (which we make precise in Section 3). This motivates our formalization of the linear representation hypothesis: that there exists a well-behaved matrixA ∈ R d×M and aK-sparse functionz:X → R M + such that for allx ∈ X, f (x) = Az(x). We formalize this as Postulate 1. 3 Analogizing to dictionaries. Sparsity ofz(x) means that while the total number of concepts may be enormous, the number of concepts needed to describe a particular input is small (at most K). This motivates an analogy to natural language, where we callA,za dictionary in which the directionsa i are the words. 1 Given a “situation” represented byf(x), the idea is to write it as the combination of a small number of words (say, at mostK). Thenz i specifies in what situations to use the directiona i (it can be viewed, therefore, as providing a “definition”). In the setting here,z i also specifies “how much” the word should be used. We callMthe dictionary size (the total number of features) andKthe description length (the maximum number of features needed to describe an input). Existing work has focused on understanding when dictionaries can be recovered from data for a fixed dictionary size and description length [30–34]. In contrast, the results we introduce here are each investigations of the broader space of potential dictionaries. Are dictionaries unique when size and description length can vary? What does an optimal dictionary look like when you restrict its size? What about when tailored to a particular distribution of situations? Deriving atomicity and a rigidity property. From the postulate, we can derive the atomic property. Theorem 1 considers a “competing” dictionary (B,u): a well-behaved matrixB ∈ R d×M ′ and K ′ -sparse function u : X → R M ′ + satisfying f (x) = Bu(x)(4) for allx ∈ X. Then the result shows that ifzis sufficiently rich (exploring each combination of features), then ifB,uis Pareto-efficient in dictionary size and description length, then either it is equal toA,zorK ′ ≥2K(i.e., it sometimes uses twice as many features to describe an input). Theorem 1 shows that if a sparse system of representation exists, and the space being described is sufficiently rich, then this sparse system is “rigid.” To get an intuitive sense of this statement, we draw on the analogy to natural language dictionaries, from which we can state the following informal analog of Theorem 1. Consider a dictionary ofM words and a set of situations that can each be exactly described by K words. Suppose further that there exist situations spanning each combination of words. Then any dictionary using one fewer word—and that can still perfectly describe each situation—must for some situation use twice as many words. (It can be informative to try to play this “vocabulary construction game” yourself. We return to this game in a moment.) From theory to testable predictions. Up to this point, we have established a pretty but (thus far) hypothetical picture of the underlying structure of language model representations. How can we know if this picture is a useful description of reality? A natural approach is to try and find the hypothesized atoms. To do this, we can try to apply sparse dictionary learning [15,29,32,35] to find a matrix ˆ A and sparse function ˆz such that f (x)≈ ˆ Aˆz(x).(5) But major challenges arise in practice. First, the true number of atomsMis unknown (and likely very large). Second, the learned dictionary is shaped by the specific training data. Finally, given a learned dictionary, how are we to know that we have recovered the true atoms? 1 Of course, we are drawing from a long line of work on dictionary learning here and throughout [29]. 4 The recovery principle. There is a powerful mathematical consequence of Postulate 1 that helps resolve these issues. Theorem 2 shows that even sparse dictionaries of sizem < Mthat are trained on idiosyncratic (but “well-behaved”) distributionsDapproximately recover a subset of atoms. In particular, as we increase the size of a sparse dictionary, we recover a growing prefix of the most prevalent atoms in the distribution. Note that the sparsityKmust be chosen appropriately to get this guarantee. 2 Notice how the recovery principle implies testable implications. First, by training larger SAEs on a fixed distribution, we should see a growing set of “persistent features”—i.e., features in smaller SAEs that appear in every larger SAE. Second, even if we train SAEs on different data distributions, we should expect to see a set of shared features—the atomic features that are most prevalent in both distributions. This can be contrasted with PCA or SVD, in which learned directions are highly dependent on the data distribution. (SVD had been suggested in the past as a “solution to Plato’s problem” [36], but a primary objection was the sensitivity to distributions; SAE features, under the atomic theory, overcome this objection.) The recovery principle and its implications are further illustrations of the rigidity of sparse systems of representation. Intuition. Returning to the vocabulary construction game, consider the goal of constructing a dictionary of limited size to best describe situations arising in your daily life using at mostKwords per situation. (Perhaps, you were confined to tweeting.) You are allowed to invent new words (e.g., referring to combinations of concepts that appear especially frequently for you). The recovery principle says that you would not pick entirely new, invented words; rather you would adopt many existing words—and in particular, the existing words that are most relevant for your own daily life. These words are retained even if you are allowed a larger dictionary. To give a concrete example, suppose that each day you looked outside your window and saw a tree by a sidewalk. Perhaps, in extremely limited vocabularies, you would define a word “sidewalktree.” This would save you budget in dictionary size and in description length. But you would be unable to describe a “tree” or “sidewalk” on their own. If you wanted to describe an isolated tree, you would need a word that means “nosidewalk” and call it a “nosidewalk” “sidewalktree,” in which case you have now used just as many words and more description length than if you had words for “tree” and “sidewalk.” In this way, there is pressure towards including the atoms “tree” and “sidewalk” in a dictionary. (To actually prove the recovery principle requires some more sophisticated reasoning.) In a sense, of course, humanity has been playing the vocabulary game for millenia. Different societies have come up with languages of different lengths to describe different distributions of situations. Quite strikingly, these languages have an enormous amount of overlap in the concepts given words. In linguistics, a “Swadesh list” gives concepts that appear in essentially all languages, including concepts like I, you, five, short, knee, to drink, to fly, to count, river, sky, red, green, year, and round (among many others). The recovery principle offers some reasoning for this—and hints at a formal notion of “universal concepts.” Recovery of hierarchical features. The recovery principle shown in Theorem 2 makes an assumption that the training distribution is well-behaved: that for every pair of featuresz i ,z j , there exist inputs for whichz i is active butz j is not (and vice-versa). In Theorem 3, we show that the recovery principle continues to hold (with an adjustment) for distributions where activations are hierarchically correlated (where a parent feature is active whenever a child feature is active). The 2 To mix metaphors, the recovery principle suggests that training larger SAEs is something like building larger particle accelerators: by scaling properly, we expect to be able to recover more and more fundamental particles (here, atoms). 5 result implies that in sufficiently large SAEs, we expect to simultaneously recover parent features and child features. This prediction is especially interesting when compared to existing empirical observations in which a broad parent feature that is recovered in a smaller SAE “splits” as SAE size increases. Under our theory, the expected behavior is that as the size of the SAE increases even further (with sparsity chosen appropriately), we should expect the parent feature to once again be recovered. 3 Theoretical Results We now mathematically present the atomic theory starting from a single postulate and deriving the testable predictions P1-P3 as consequences. It will be useful to first establish a formal definition of a dictionary, which is the primary object we study throughout. Definition 1. An (m,k,γ)-dictionary is a pairB ∈ R d×m ,u:X → R m + whereuisk-sparse and B ∈ R d×m has unit-norm columns and satisfies the (γ,2k)-lower restricted isometry property: that is, for all 2k-sparse v ∈ R m , ∥Bv∥ 2 ≥ γ∥v∥ 2 . 3 Recall thatmis the dictionary’s size andkthe dictionary’s description length. The lower restricted isometry property establishes the key property thatu(x) can be recovered fromBu(x). Indeed, ifBu(x) =Bu ′ for somek-sparse vectoru ′ , then∥u(x)− u ′ ∥ ≤ 1 γ ∥Bu(x)− Bu ′ ∥ 2 = 0, sou(x) =u ′ . The dictionaries we study are overcomplete (m > d) and sparse (k < d). In what follows, we takeγ ∈(0,1) to be a fixed constant and refer to an (m,k,γ)-dictionary as simply an (m,k)-dictionary. We can then succinctly state our foundational postulate about f. Postulate 1 (The Linear Representation Hypothesis). There exists an (M,K)-dictionaryA,zsuch that f (x) = Az(x)(6) for all x∈ X. We refer toA,zas the atomic dictionary, with atomic directionsa 1 ,a 2 ,· ,a M and atomic features z 1 ,z 2 ,· ,z M . Our first task is to justify calling these features atomic. Past work has shown that ifzis sufficiently “rich,” thenA,zis the unique dictionary with dictionary size and description length exactly equal toMandK[30,31,34]. Here, we will establish a notion of uniqueness that is quite a bit broader. We say that an (m,k)-dictionary is efficient if, compared to the atomic dictionary, any one of the following three conditions holds: (a) it has the same dictionary size and description length (i.e., m = M,k = K); (b) it has smaller description length (i.e., k < K); (c)it has smaller dictionary size and less than twice the description length (i.e.,m < Mand k < 2K). Now define richness as follows. ForS ⊆[M] with|S|=K, letE S =spane i :i ∈ Sand Z S =z(X)∩ E S (the set of feature activations with supportSrealized byX). We callz K-rich if Z S is not contained in any union of a finite number of proper subspaces of E S . 3 For d = Ω k,γ (logm), lower RIP holds with high probability for random Gaussians [27] 6 Theorem 1 (Unique Efficient Dictionary). SupposezisK-rich. If an efficient dictionaryB,u satisfies f (x) = Bu(x)(7) for all x∈ X, then B,u = A,z up to a permutation of indices. Therefore, no dictionary that exactly describesfcan have smaller description length thanA,z, and any smaller dictionary that exactly describesfmust have double the description length. One way to view this result is that if a set of inputs can be described by a sparse dictionary, then this sparse dictionary is rigid. 4 The main challenge of the proof is to show thatk <2K=⇒ m ≥ M. The key idea is to construct a bipartite graph between directions in the two dictionaries, wherea i andb j share an edge ifb j is used in the reconstruction ofa i (we will first have to show thata i has ak-sparse representation inB). Then, we can show that ifk <2K, Hall’s condition is satisfied for each subset ofAdirections of size at most 2K; moreover, no subset of at mostK Adirections has a neighborhood of size at least 2K. We then show that these two conditions imply Hall’s condition to hold globally. (Interestingly, this appears to be a novel sufficient condition.) Therefore, there exists a one-to-one matching including all A directions, showing that m≥ M. The recovery principle. We now establish the recovery principle: that even undersized SAEs trained to approximate idiosyncratic distributions of inputs recover atoms. This gives testable predictions P1 (features in small SAEs persist in larger SAEs) and P2 (SAEs trained on different data learn overlapping features). The recovery principle concerns dictionaries that (1) are undersized (i.e.,m < M) and (2) only approximately describefon some distributionDoverX. The recovery principle shows conditions under which such dictionaries still recover atoms. Define the dictionary learning objective L D (B,u) := E x∼D ∥f (x)− Bu(x)∥ 2 2 (8) over the reference class of (m,K)-dictionaries. Notice that dictionaries in this reference class have description length matching the atomic dictionary; we show in simulations in Section G that choosing appropriatekis important, but that there is some flexibility. LetL ∗ D (m) be the optimal loss over this class. Taking a fixed constant C 0 > 1, let ˆ A (D,m) , ˆz (D,m) be an (m,K)-dictionary satisfying L D ( ˆ A (D,m) , ˆz (D,m) )≤ C 0 L ∗ D (m).(9) Conceptually, this is the kind of dictionary that a sparse autoencoder can learn in practice, obtaining loss within a constant factor of optimal. We are interested in if and when ˆ A (D,m) , ˆz (D,m) recovers atomic features and directions. Formally, we define recovery as follows: • Direction recovery: ˆ A η-recovers a i if there exists j ∈ [m] such that ⟨ˆa j ,a i ⟩ > η. •Activation recovery:ˆz(η,D)-recoversz i if there existsj ∈[m] and a thresholdτsuch that F1 D (1z i > 0,1ˆz j > τ) > η. We show that recovery can happen as long as the distributionDis well-behaved. As an example of why some kind of assumption onDis necessary, notice that if two atomic featuresz i ,z j were exactly co-occurring on a distribution, then it would not be possible to disentangle them. 4 A result by Sitharam et al.[37]establishes a related rigidity result: that one cannot achieve a constant fraction of dictionary size without description length blowing up to be on the order of embedding dimension [37]. We state this result as Theorem 4 and give an alternate proof using Sard’s theorem. 7 This motivates the following definition. A distributionDoverXis (ρ,c − ,c + )-regular if the following conditions hold forx ∼ D.First, as a simplification, letsupp(z(x)) have sizeKfor all inputs. Leth S : (0,1) K → R + be the density ofz S (x) conditional on the eventsupp(z(x)) =S. Then the two conditions are • Pr[j /∈ supp(z(x))|i∈ supp(z(x))]≥ ρ for all pairs i,j ∈ [M ] with i̸= j. • c − ≤ h S (w)≤ c + for all w ∈ (0, 1) K . The idea of the recovery principle, then, is that ifDis regular, then undersized sparse dictionaries can recover the most prevalent atoms in D. GivenD, setp i :=Pr[i∈ supp(z(x))], the probability that an atomiis active for a given input. Chooseσ: [M]→[M] such thatp σ(1) ≥ p σ(2) ≥ · ≥ p σ(M) ,determining the order of prevalence. Intuitively, we may expect these probabilities to follow a power-law distributionp σ(i) ∝ i −α . Define T(m) = P M j=m+1 p σ(j) , which we can call the tail mass beyondm. Then Theorem 2 below says that an atom is recovered in a dictionary of sizemwhenever its prevalence exceeds the tail massT(m) by a constant factor. Theorem 2 (The Recovery Principle). LetDbe (ρ,c − ,c + )-regular. Then for everyη <1, there exists a constant C not depending on m or M such that for all i∈ [M ] satisfying p i ≥ CT (m),(10) ˆ A D,m η-recovers a i and ˆz D,m (η,D)-recovers z i The result guarantees recovery of atoms that appear more frequently than the tail mass beyond them-th most frequent atom. For example, if atom prevalences follow a power-law distribution with α >1, the result guarantees that at least the Ω(m 1−1/α ) most prevalent atoms are recovered. In Section G, we show that in simulations, a much larger fraction of atoms can be recovered—suggesting that the bound shown here is not tight (one reason, perhaps, is that the bound here applies for any learned dictionary with low enough loss, and does not take advantage of the structure of optimal learned dictionaries). It would be exciting to prove stronger versions of the recovery principle. To provide some intuition for the result, we sketch the proof. The main lemma shows that if there exists an atomic supportS ⊆[M] such that the learned dictionary approximates representations that lie inspan(A S ), then there must exist a learned supportT S ⊆[m] such thatspan( ˆ A T S )≈ span(A S ). This is non-obvious, since in principle, points inspan(A S ) may have different optimalK-sparse supports in ˆ A. An initial observation is that if we can show that a single learned support is being used to re-construct a large enough subset ofspan(A S ), then the span of this support must be aligned withspan(A S ). This approach does not work immediately, since there are m K possible supports, so each learned support may only be used for a vanishing fraction of the span asmincreases. A key insight is that the lower restricted isometry property implies that optimal supports do not change too much for nearby points, allowing us to iteratively identify a large enough ball in which a single learned support is used. From this lemma, we can then intersect recovered supports to show the recovery of individual atoms. Finally, any sufficiently optimal dictionaries—when the tail mass is small—must recover atomic supports spanned by atoms in the head of the distribution. Predictions P1 and P2 follow as corollaries. Both concern the existence of matching features across dictionaries. For matricesUandV, we defineμ t (U,V) to be the size of the largest one-to-one matching between columns ofUandVsuch that matches have cosine similarity at leastt. Let μ t (U;V 1 ,V 2 ,· ,V s ) be the maximum number of columns ofUthat can be simultaneously matched 8 to everyV 1 ,V 2 ,· ,V s . Then fort <1, there exists a constantC t such that forR D,t (m) :=maxr ∈ [M ] : p σ(r) ≥ C t P M j=m+1 p σ(j) the following results hold. Corollary 1 (P1: Stability Across Size). For a regular distributionDand dictionary sizesm 1 ,...,m s ≥ m, μ t ˆ A (D,m) ; ˆ A (D,m 1 ) ,..., ˆ A (D,m s ) ≥ R D,t (m). That is, there exists a persistent set of features across all dictionaries of size at least m. Corollary 2 (P2: Stability Across Data). For any regular distributionsDandD ′ on the same atomic representation, μ t ˆ A (D,m) ; ˆ A (D ′ ,m) ≥ σ([R D,t (m)])∩ σ ′ ([R D ′ ,t (m)]) . Here,σ([R D,t (m)])∩ σ ′ ([R D ′ ,t (m)]) is the set of atoms in both distributions’ guaranteed prefixes. So dictionaries trained on different distributions have shared features—the atoms prevalent in both distributions. The hierarchical recovery principle. To establish P3 (that sufficiently large SAEs recover both parent features and child features), we show that the recovery principle holds under distributions where atomic feature activations have hierarchical correlation: i.e., when the activation of a child atom implies the activation of a parent atom. This violates the regularity condition onDused in Theorem 2. We consider a setting withNfamilies, each of which has one parent atom and two child atoms. It is possible to generalize to more than two child atoms. We letM= 3NandK= 2L, and re-index the atoms asi,i1,i2 fori∈[N]. Here, the parent featurez i is always active when the child features z i1 orz i2 are active. We then establish a definition of hierarchical regular distributions. Under this definition, parent features satisfy the standard regularity assumptions, exactly one child feature activates at a time, and the space of parent-child activation pairs is well-explored. The full definition is given in Section B. We then define analogs of quantities used in the regular recovery principle. For a givenDand a familyi ∈[N], letp i =Pr[i ∈ supp(z(x))] andσ: [N]→[N] sort the families in monotonically decreasing order of prevalence according to occurrence of the parent feature (soσ(1) is the most prevalent family). Define T (n) = P N j=n+1 p σ(j) , the tail mass beyond the n-th family. Then Theorem 3 below shows that an analog of Theorem 2 holds in hierarchical settings for activation recovery. In other words, we should expect to be able to recover features that activate in the presence of the parent, and also features that activate in the presence of each child, yielding P3. Theorem 3 (The Hierarchical Recovery Principle). LetDbe hierarchical regular. Then for every η <1, there exists a constantCnot depending onmorMsuch thatˆz D,m (η,D)-recovers the family of features z i ,z i1 ,z i2 for all i∈ [N ] satisfying p i ≥ CT j m 3 k .(11) The following identity provides useful intuition. Ifz j (x)>0 =⇒ z i (x)>0, then for allλ≥0 we can re-write z i (x)a i + z j (x)a j = ˆz i (x)ˆa i + ˆz j (x)ˆa j ,(12) 9 whereˆz i (x) =z i (x) +λz j (x), ˆa i =a i , ˆz j (x) =z j (x),andˆa j =a j − λa i . 5 This preserves sparsity and non-negativity ofu. More importantly, bothˆz i andˆz j activate on exactly the same examples asz i andz j . Therefore, even though atomic directions and features are not strictly unique under hierarchical correlation, alternative parameterizations are still equivalent in terms of when each feature is active. Mesoscale feature splitting. The prediction of simultaneous recovery of both parent and child features at large SAE sizes seems in tension with empirical results that have demonstrated feature splitting and absorption [22–24]. P3 suggests that as SAE size continues to increase, we should expect lost parent features to return. We give a “molecular” model of this phenomenon in Section 5, in which small SAEs learn a family molecule, medium SAEs learn individual parent-child molecules (this is the regime where a learned feature matching the parent’s activation is lost), and large SAEs learn an individual parent atom and individual child atoms. This molecular model can explain further phenomena, such as polysemanticity and semantic “clusters” of SAE directions. 4 Experiments: Testing P1, P2, and P3 We now test the predictions P1, P2, and P3 derived from Postulate 1. We study two large embedding models, gemini-embedding-2 [38] and llama-embed-nemotron-8b [39], which give representationsf(x) of dimensiond= 3072 andd= 4096, respectively. We traink-sparse autoencoders [35] on embeddings derived from 90M text inputs, comprising 13 datasets used to fine-tune llama-embed-nemotron-8b. An SAE of sizemand trained with sparsitykproduces a dictionary ˆ A ∈ R d×m , ˆz:X → R m , where ˆ A=W dec is the SAE decoder andˆz(x) =TopK k (ReLU(W enc (f (x)− b dec )))are the hidden layer activations. We train SAEs of sizesmin powers of two ranging from 512 to 131,072, with k= 32,32,32,32,64,64,64,128,128 respectively. (Section D reproduces results withk= 128 fixed.) Figure 2 displays our main results. Details of our experimental setup are given in Section C. Extended analyses for P1, P2, and P3 are collected in Sections D to F, respectively. Computing matching features. To test P1 and P2, we need to compute matches across dictio- naries. We use the following metrics. Given a bipartite graph, a maximum matching is a one-to-one matching with the largest number of matches. The exclusive matching is the one-to-one matching consisting of all edges whose endpoints have only one neighbor. Given dictionaries ˆ A, ˆzand ˆ A ′ , ˆz ′ ,we draw edges between directions that have cosine similarity at leasttand edges between features that have Pearson correlation at leasttover a specified distribution. We then compute the maximum and exclusive matching. Given a matching criterion (maximum or exclusive), we compute a matching independently between the source dictionary and each larger dictionary. A direction or feature is persistent if it is matched in every comparison. We use a threshold oft= 0.7 as our default, following Paulo and Belrose [20]. P1: Persistent features across size. We start by testing P1, which says that many features in smaller SAEs have matches in all larger SAEs. Many directions and features have matches in all larger SAEs. Under independent maximum matching at thresholdt= 0.7, in SAEs of size 4,096, 72.6% of Gemini directions and 69.4% of Nemotron directions are persistent (Figure 2a). 5 This identity is studied by Chanin et al.[22]as a potential source of feature absorption; an important distinction in our assumptions is that we allow child feature activations to be larger than the parent, meaning thatˆz i = z i (x) + λz j (x)≥ 0 =⇒ λ≥ 0. 10 Figure 2: Testing P1, P2, and P3—the predictions of the atomic theory. (a) P1: Persistent features across size. Number and proportion of directions with matches in every larger SAE. Similarity thresholdt= 0.7. About 70% of 4K SAE directions are retained in size-8K, 16K, 32K, 65K, and 131K SAEs. 5121K2K4K8K16K32K65K SAE width 0 5,000 10,000 15,000 20,000 # persistent directions gemini nemotron 5121K2K4K8K16K32K65K SAE width 0% 20% 40% 60% 80% 100% % persistent directions (b) P2: Stability over data distributions. Proportion of directions shared between SAE and PCA dictionaries trained on different distributions. Similarity thresholdt= 0.7. We use absolute cosine similarity for PCA directions. 26%-59% of SAE feature directions have matches, compared to mostly 0% and at most 15% for PCA. full wiki no_wiki rand1rand2 full wiki no_wiki rand1 rand2 0.470.380.500.48 0.470.260.480.41 0.380.260.390.42 0.500.480.390.42 0.480.410.420.42 SAE (gemini) full wiki no_wiki rand1rand2 0.490.380.530.54 0.490.290.590.50 0.380.290.400.49 0.530.590.400.52 0.540.500.490.52 SAE (nemotron) full wiki no_wiki rand1rand2 0.080.010.070.07 0.080.010.150.04 0.010.010.010.03 0.070.150.010.05 0.070.040.030.05 PCA (gemini) full wiki no_wiki rand1rand2 0.000.000.000.00 0.000.000.030.00 0.000.000.000.01 0.000.030.000.01 0.000.000.010.01 PCA (nemotron) 0.0 0.2 0.4 matching feature proportion (c) P3: Hierarchical recovery. Mean held-out test F1 by hierarchy level and SAE width for Prefix, Cities, and GBIF. Each curve is an unweighted mean over category–model pairs, pooling Gemini and Nemotron. Average recovery is highest across all levels at sizes 65K and 131K. 512 2,0488,192 32,768 131,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Mean held-out test F1 Prefix 1-letter 2-letter 512 2,0488,192 32,768 131,072 SAE width Cities Continent Country 512 2,0488,192 32,768 131,072 SAE width GBIF Kingdom Phylum Class Order Family Genus 11 By activation similarity, 59.7% of Gemini features and 66.4% of Nemotron features are persistent (Figure 7). Section D shows results under exclusive matching and under different thresholds. We also compare feature prevalence across SAE sizes. Persistence suggests that larger SAEs do not learn strictly more granular features. We find that larger SAEs learn a similar number of coarse features as smaller SAEs while learning a longer tail of granular features; in contrast, KMeans clusters become strictly smaller as the number of clusters increases (Figure 10). Consistent with recovering prevalent atomic features, more prevalent features are more likely to be persistent (Figure 12). P2: Shared features across training data. We test P2, which says that SAEs trained on different data learn overlapping features. We use four additional data splits:wiki,no_wiki,rand1, andrand2. The pair (wiki,no_wiki) contain separate datasets (wikicontains datasets that are built using Wikipedia data, andno_wikicontains the datasets that are not). The pair (rand1,rand2) contain separate datasets as well, but the partition is chosen randomly rather than by content type. None of the four data splits contain the multilingual dataset MIRACL. We train SAEs with m= 16384 andk= 64 on each distribution. We compare to PCA baseline dictionaries, consisting of thedprincipal components fitted to each distribution. To compute matchings between PCA directions, we use absolute cosine similarity, since the signs of principal components are arbitrary. Figure 2b shows the proportion of features shared across pairs of different dictionaries trained on different datasets fort= 0.7. SAE dictionaries have many cross-dataset matches (between 26% and 59%), while PCA dictionaries have substantially fewer (at most 15%, median 1%). The lower matching rates in the PCA dictionaries suggest that the high amount of overlap in SAE dictionaries is not explained only by similarity in the underlying data distributions. Figure 14 also shows that SAEs have many cross-distribution matches based on activation similarity (22-61%). Under exclusive matching, SAE dictionaries continue to have many matching directions (25-57%) whereas the PCA match proportion remains at most 15% (Figure 13). Figure 15 shows that features with greater prevalence in both distributions are more likely to be selected in direction matching, consistent with the recovery of overlapping prevalent atoms. P3: Hierarchical feature recovery. We test P3 by studying how well SAEs of different sizes recover a ground-truth set of hierarchical features defined by three datasets:prefix,cities,gbif. For each ground-truth binary featurez ∗ ℓ :X →0,1defined over inputs from each dataset, and for each SAE size from 512 to 131K trained onfull, we report the test F1 scoreF1 E test (z ∗ ℓ , 1ˆz j ≥ τ), wherej ∈[m] andτ >0 are selected to maximize the same score underE train . HereE train andE test are the empirical distributions on the train and held-out test splits of the labeled dataset. Dataset and experiment details are given in Section C.5. Figure 2c shows that across levels of the hierarchy, mean test F1 generally improves with size and is highest at width 65K or 131K. Figure 18 confirms that this corresponds to increasing simultaneous recovery of parent and child features. We also find examples of mesoscale feature splitting. Figure 3a shows that F1 recovery of the parent ‘a’ prefix feature decreases as SAE size increases initially (splitting), but increases again at even larger sizes. (Figure 19 shows all 25 Gemini letter prefixes with eligible two-letter children, including ‘a’ and excluding ‘x’.) However, this effect disappears when we increase the sparsity of smaller widths to match the largest widths, suggesting that feature splitting may primarily be the result of over-constrained sparsity. See Section 5 for more discussion. In addition to testing how well SAEs recover ground-truth hierarchies, we also extract hierarchical structure directly by examining correlation in feature activations over the corpusfull(Section F.4). We find 114,762 parent-child pairs in the 131K gemini SAE. Figure 3b shows examples of extracted three-level hierarchies, with top-level features “Food” and “Music.” Example lineages include “Food” → “Potatoes” → “Hash browns / rosti” and “Music” → “Jazz” → “John Coltrane.” 12 512 1024204840968192 163843276865536 131072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Test F1 a-prefix (gemini) a mean a-az mean 512 1024204840968192 163843276865536 131072 SAE width a a ab ac ad ae af ag ah ai ak al am an ap ar as at au av aw ay az Prefix a-prefix probe recall (gemini) 0.0 0.2 0.4 0.6 0.8 1.0 Recall of best a-prefix SAE feature (a) Mesoscale feature splitting. Level 1 Level 2Level 3 FoodCheeseCheddar; Mozzarella / related cheeses; Blue cheeses; Gruyère / Emmental; Brie / Camembert PotatoesPotato cultivars; Mashed potatoes; Baked potatoes; Hash browns / rösti; Tater tots Fast foodBurger King; Wendy’s; Jack in the Box; Hardee’s; McDonald’s burgers Rice dishesBiryani; Nasi goreng; Fried rice SausagesChorizo; Salami; German sausages; Hot dogs Nutritional information Fast-food calorie questions; Calorie-breakdown template; Percent Daily Value text; Vitamins / minerals Music JazzCharlie Parker / bebop; John Coltrane; Miles Davis; Mahavishnu / John McLaughlin; Marsalis family Hip-hop / rapN.W.A; Eminem; Snoop Dogg; Tupac Shakur; Lil Wayne Country musicGeorge Jones; Rascal Flatts; Patty Loveless / Vince Gill; Hank Williams Rock / metal Hard-rock musicians / bands; Nu / alternative metal; Power / symphonic metal; Doom / stoner metal Electronic / danceDetroit techno cluster; Trance mixes / compilations; Basement Jaxx; deadmau5 The BeatlesPlease Please Me / early Beatles; Hey Jude; Let It Be / Get Back; Abbey Road / late Beatles (b) Learned feature hierarchies. Figure 3: Hierarchical feature recovery. (a) Top: parent and child F1 for a-prefixes. Parent recovery follows a U-shape. Bottom: recall of the selected a-prefix feature, broken down by prefix. (b) Extracted Food and Music hierarchies in the Gemini 131K SAE. Activating examples for each feature are given in Tables 8 and 9. 5 A Molecular Model We now sketch a molecular model that helps explain several additional empirical phenomena: polysemanticity, feature splitting, and semantic clusters of directions [24]. In the model, “molecules” are learned SAE directions that are linear combinations of atomic directions. The recovery principle says that molecules in small SAEs eventually split into stable atoms in larger SAEs. A prediction, then, is that since the above empirical phenomena stem from molecules, they disappear at sufficiently large widths. The results and explanations here are intended to be illustrative—we hope that future work formalizes and systematically tests the molecular extension of the atomic theory. The intuition for molecules can be conveyed with a minimal example. If two atomic features z i (x) andz j (x) are heavily correlated overD, then it can be efficient for ˆ A(trained to minimize L D ) to mix the atomic directionsa i anda j into a single learned direction, saving budget in terms of dictionary size (and also description length). The direction can be visualized as follows: i j (13) In the empirical results that follow, we use learned directions and features in the size-131K gemini SAE as proxy atoms. We then decompose learned directions in smaller SAEs as linear combinations of proxy atomic directions, giving proxy molecules. Details are given in Section H. 13 Table 1: A polysemantic “transform”-related feature in the 4K gemini SAE, represented as a molecule of 131K features. It activates on mentions of “transform” as well as strings like “are autobots stronger than decepticons” (referencing the Transformers franchise). Examining activating examples in the table, the 131K features (treated as proxy atoms) seem comparatively monosemantic (e.g., representing the Transformers franchise). More examples for each atom are given in Table 11. 8574 114331 68236 72509 97729 The word family “transform” and “transformation” Transformers franchise Change and its word forms/translations Electrical transformers Composing transformations and their matrix representations z 4K 3290 Textz 131K 8574 z 131K 68236 z 131K 114331 z 131K 72509 z 131K 97729 0.387 This is a list of transforms in mathematics.0.2520.0870.0000.0670.014 0.330 what is the importance of data transformation?0.2550.1020.0000.0090.000 0.318A transformation group is a group G that acts on a set X by bijections. In other words, for each element g in G, there ... 0.2500.0470.0000.0000.000 0.285 A non-linear transformation is a mathematical function that doesn’t fol- low the rules of linearity. In simple terms, a linear ... 0.2110.0680.0000.0000.014 0.257 To transform the given sum, utilize the decomposition dfrac 1(k(k+1)(k+1)!)=left(dfrac 1(kcdot k!)-dfrac 1((k+1)(k+1)!) ... 0.2170.1240.0000.0000.000 0.163 The Metamorphosis (German: Die Verwandlung ) is a novella written by Franz Kafka which was first published in 1915. 0.0850.0780.0000.0000.000 0.148 what is ctrl t in photoshop?0.0790.0430.0000.0000.000 0.204 ... voice actors Peter Cullen and Hugo Weaving voice Optimus Prime and Megatron respectively. 0.0300.0000.2100.0000.000 0.169 are autobots stronger than decepticons?0.0000.0000.2420.0320.000 0.338 A transformer is a device that transfers electrical energy from one circuit to another by magnetic coupling. 0.1650.0550.0360.2990.000 0.186 why crgo core is used?0.0550.0220.0000.1770.000 0.168 Translation: 15 units left, 3 units down; Reflection: across x-axis; Scale: along y-axis by a factor of 2. 0.0750.0610.0000.0000.063 0.1521. Reflect about the x-axis: - A(1, 5) → A'(1, -5) - B(0, 1) → B'(0, -1) 2. Translate vertically by 3: - A'(1, -5) → A′(1, -2) - B'(0, -1) → B′(0, 2) 3. Rotate about the origin... 0.0370.0750.0000.0000.097 Selective examples for individual atoms 0.159 At transform plate boundaries plates grind past each other side by side. This type of boundary separates the North American plate from the Pa- cific plate along the San Andreas fau... 0.0950.0220.0000.0000.000 0.000 how do i change my azure subscription?0.0000.2390.0000.0000.000 0.202 who does optimus prime fight in transformers autobots0.0000.0000.2700.0000.000 0.172 How should I wire a center-tapped transformer with input voltages of 0V and 230V and output voltages of 0V, 20V, 0V, and 20V? Can I safely connect the two middle outputs together... 0.0490.0000.0000.1990.000 0.060 To represent the combined rotation of a 3D point (x, y, z), you can mul- tiply the rotation matrices corresponding to each axis in the order of rotation: first Z-axis, then X-axi... 0.0000.0000.0000.0000.102 14 dict. size iˆa i decoder molecule recall on ah recall on ab recall on ap recall on as 512 108 0.91a ∗ a + 0.11a ∗ as +0.08a ∗ ab + 0.03a ∗ ap +0.02a ∗ ah a as ab ap ah 1.0000.9620.8770.993 2048 1457 0.92a ∗ a + 0.10a ∗ ah a ah 1.0000.2360.3390.388 1794 0.85a ∗ ab + 0.36a ∗ a ab a 0.0000.8730.0030.000 413 0.84a ∗ ap + 0.34a ∗ a ap a 0.0000.0030.8870.000 183 0.79a ∗ as + 0.36a ∗ a as a 0.0290.0030.0000.783 131K 38618 1.00a ∗ a a 1.0000.9500.9320.980 121657 1.00a ∗ ah ah 0.8710.0000.0000.000 100965 1.00a ∗ ab ab 0.0000.7490.0000.000 120576 1.00a ∗ ap ap 0.0000.0030.7980.000 32170 1.00a ∗ as as 0.0290.0000.0000.765 Table 2: Empirical a-prefix feature splitting viewed in terms of molecules. A family molecule splits into parent-child molecules into atoms. In the middle regime, no molecule corresponds to a feature that activates on the full family (feature splitting). Such a feature exists in the small and large regimes. Polysemanticity. Some SAE features are polysemantic, mixing loosely related concepts. In the molecular view, this is due to a direction that is a mixture of atomic directions. Table 1 gives a direction in the 4K gemini SAE written as a molecule, with atoms corresponding to the word transform, as well as the Transformers franchise, electrical transformers, mathematical compositions of transforms, and the concept of change. Intuitively, these are mixed together in the small SAE since they are correlated. The table shows how texts that activate the polysemantic 4K SAE feature also activate the proxy atomic features. For example, “are autobots stronger than decepticons?” activates the original feature, as well as the Transformers franchise feature in the 131K SAE, but not the other proxy atomic features. We also find that many inputs that activate the broader “change” feature in 131K do not activate the polysemantic 4K feature. This is an example of asymmetry between the molecular representation of learned directions and the activation pattern of the corresponding feature. This plays a central role in the explanation for feature splitting below. Feature splitting and absorption. Consider a family with one parent atom and two child atoms, and suppose that the parent feature activation is highly correlated with the child feature activation. Concretely, we can consider a pet atom, along with a dog atom and cat atom. Then consider what happens when a different number of features are “budgeted” to this family. •If only one feature is budgeted, then the learned direction is a mixture of the parent direction 15 with small contributions from each child direction (this is the first un-normalized principal component). pet cat dog (14) This direction is used in the reconstruction of every input that activates the family. Therefore, its corresponding feature has the same activation pattern as the parent atom. • If two features are budgeted, then the optimal learned directions include one for each parent-child pair; each is given by a sum of the parent direction and child direction. pet cat pet dog (15) Each molecule is used only for inputs that activate the specific child. Therefore, there is no learned feature that activates for all members of the family (giving “feature splitting”). Notice how this solution also minimizes description length (k). • If three features are budgeted, then the learned directions can match the actual parent and child atoms (up to the re-parameterization given in (12)). pet cat dog (16) In particular, as shown in Theorem 3, there is a learned feature with an activation pattern corresponding to that of the parent and each child. This gives a theoretical explanation of mesoscale feature splitting behavior that is consistent with P3. The example illustrates a second reason for feature splitting: over-restriction of the dictionary’s description length (sparsity). Table 2 illustrates this behavior in real SAEs, where we have restricted our attention to a subset of proxy atoms that correspond to one- and two-letter “a” prefixes. At small dictionary size, there is a large molecule that is active for ah, ab, ap, and as. At intermediate size, there are instead molecules that mix the broad parent atom with the individual ah, ab, ap, and as child atoms. None are active for the whole family, and so the feature “a” appears to disappear (though its direction is still included in the molecules). Finally, at the largest size, individual atoms are recovered, including a feature that activates on the whole family. Correlated directions forming semantic clusters. Various papers report seeing clusters of learned SAE directions that are correlated. As a concrete example, in Anthropic’s SAEs trained on Claude 3 Sonnet, Templeton et al.[24]find that “around the Golden Gate Bridge feature 34M/31164353 ... there are features corresponding to particular locations in San Francisco such as Alcatraz and the Presidio” [24]. Our model suggests that these directions are molecules rather than atoms: the directions are correlated since they share a common atom (corresponding to the shared geographical region). Indeed, Templeton et al.[24]note a broader correlated San Francisco direction in a smaller SAE, consistent with the feature splitting view presented above. In underbudgeted regimes (in either dictionary size or description length) learning these molecules can be efficient. Our results suggest, however, SAEs with enough budget should learn individual atoms, rather than molecules. Then, at large budgets, we should see the observed correlation in directions decrease substantially. 16 We find evidence consistent with this prediction. Figure 26 compares the decoder directions selected as a-prefix probes in our 32K and 131K gemini SAEs. The median cosine over the 231 unique off-diagonal pairs is 0.144 at width 32K and 0.008 at width 131K. Direction correlation between extremely related features (e.g., the ‘ap’-prefix and the ‘ad’-prefix) can be very small or even negative. Given the large size of the SAEs trained by Anthropic, our model predicts that these SAEs are underbudgeted in description length (k < K). 6 The theory and observations here are consistent with work by Chanin et al.[22], who show that mis-specifiedkleads to correlation in learned feature directions, and suggest using orthogonality in learned features as a way to tune sparsity. A broad implication is that many undesirable aspects of learned SAEs are explained by molecules. So problems can be solved by scaling SAEs, with appropriate sparsity [22], so that molecules split into their constituent atoms. 6 Discussion and Further Results We developed an atomic theory of language model representations, deriving three testable predictions from the LRH and validating these predictions on SAEs trained on two large embedding models. (A natural next step is to test these predictions on other representations—e.g., residual streams.) Here, we discuss further applications of the atomic theory and connections to related work, alongside additional experiments. The Recovery Principle. The recovery principle suggests the promise of scaling standard SAEs. In particular, large SAEs should be able to simultaneously recover stable coarse and granular features, reducing the need for incorporating hierarchical structure directly (cf., [23,40–43]). Two aspects of the recovery principle are worth further investigation. First, Theorem 2 gives fairly weak guarantees in terms of the number of recovered atoms (e.g., a vanishing fraction of SAE sizem). In Section G, we find that SAEs trained on simulated data can recover a much larger proportion of atomic features. Second, Theorem 2 requires exact sparsityK. While our results suggest that correct sparsity specification is important, our simulations again suggest some robustness. Practically, it is important to know when sparsity is being chosen correctly [25]. Multi-dimensional, non-linear, and manifold hypotheses. Some work has noted that repre- sentations can exhibit higher-dimensional non-linear structure [44–47]. In Section I, we suggest that these observations are consistent with the atomic theory. We show that correlations in feature activa- tions alone (absent any correlation in their associated directions) are enough to produce manifold structure. Figure 27 shows that small sets of features recover manifolds on dates and hex colors. (As a “bonus,” Table 12 shows that these color features generalize across hex codes, text, images.) A rigidity principle, platonic representations, and human interpretability. A growing literature documents increasing similarity in representations learned between different AI models [48– 51]; work studying “algorithmic monoculture” has theoretically and empirically analyzed homogeneity in model outputs and its implications [52–62]. A “rigidity principle” illustrated by Theorems 1 to 3 suggests an explanation: that sparse systems of representation tend to involve the same concepts (recall, again, the vocabulary construction game). If AI models all use sparse representations, then these should be the same. Formally, we might 6 In the case of Anthropic’s SAEs, which do not use a hard sparsity constraint, the sparsity penalty is too high. 17 hypothesize that different representations factor asf=Azandf=A ′ zfor different choices of A∈ R d×M andA ′ ∈ R d ′ ×M , where the atomic feature representationz:X → R M is shared. In the language of the platonic representation hypothesis [48], one could hypothesize a set of platonic atoms. Then, as in the recovery principle, larger and larger models could be learning a growing subset of these platonic atoms, explaining increasing representational similarity. (Empirically, homogeneity seems to be increasing with model size beyond what can be explained by increasing accuracy [59,60].) In Section J, we identify matches between features in SAEs trained on gemini and nemotron. We find that size-131K gemini and nemotron SAEs have 11K matching features at Pearson similarity threshold 0.7 and 27K matching features at threshold 0.5. Example shared features (shown in Tables 13 to 24) include “mechanical bearings,” “comments in source code,” “trade tariffs and protectionism,” “trigonometric half-angle identities,” and “astragalus and milkvetch species.” The fact that SAE features trained on AI models are human-interpretable [15,18,19,24] is quite amazing, and not obvious a priori: we could have imagined AI models learning different, inscrutable representations. One hypothesis is that since both humans and AI rely on sparse representations, there is pressure towards shared concepts. Code Code to reproduce data preparation, SAE training, and experiments is available in this repository: https://github.com/kennylpeng/atomic_theory. AI Use The conceptual ideas in the paper are those of the authors. We used AI to prove the ultimate theorem statements in the paper (after proving restricted versions ourselves) and help draft appendix sections. We reviewed and edited proofs (and rewrote proofs of central claims for clarity). Lean verifications for theoretical results are given in this repository:https://github.com/nikhgarg/ atomic-features-formalization/ , with autoformalization and verification using the AppliedMod- elingLib project [63]. References [1]Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013. [2] Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746–751, 2013. [3]Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483–495, 2018. [4] Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023. [5]Belinda Z Li, Zifan Carl Guo, and Jacob Andreas. (how) do language models track state? arXiv preprint arXiv:2503.02854, 2025. 18 [6]Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681, 2023. [7]Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023. [8]Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2023. [9]Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A Chi, Samuel Miserendino, Johannes Heidecke, Tejal Patwardhan, and Dan Mossing. Persona features control emergent misalignment. arXiv preprint arXiv:2506.19823, 2025. [10]Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. Persona vectors: Monitoring and controlling character traits in language models. arXiv preprint arXiv:2507.21509, 2025. [11]Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022. [12]Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023. [13] Valérie Costa, Thomas Fel, Ekdeep Singh Lubana, Bahareh Tolooshams, and Demba Ba. From flat to hierarchical: Extracting sparse representations with matching pursuit. arXiv preprint arXiv:2506.03093, 2025. [14]Nikhil Garg, Jon Kleinberg, and Kenny Peng. How many features can a language model store under the linear representation hypothesis? In Steve Hanneke and Tor Lattimore, editors, Proceedings of Thirty Ninth Conference on Learning Theory, volume 336 of Proceedings of Machine Learning Research, pages 5358–5376. PMLR, 29 Jun–03 Jul 2026. URLhttps: //proceedings.mlr.press/v336/garg26a.html. [15]Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. [16]Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https://transformer- circuits.pub/2023/monosemantic-features/index.html. [17]Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. 19 [18]Rajiv Movva, Kenny Peng, Nikhil Garg, Jon Kleinberg, and Emma Pierson. Sparse autoencoders for hypothesis generation. arXiv preprint arXiv:2502.04382, 2025. [19]Kenny Peng, Rajiv Movva, Jon Kleinberg, Emma Pierson, and Nikhil Garg. Use sparse autoencoders to discover unknown concepts, not to act on known concepts. arXiv preprint arXiv:2506.23845, 2025. [20]Gonçalo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different features. arXiv preprint arXiv:2501.16615, 2025. [21]Thomas Fel, Ekdeep Singh Lubana, Jacob S Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Wattenberg, Demba Ba, and Talia Konkle. Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models. arXiv preprint arXiv:2502.12892, 2025. [22]David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Satvik Golechha, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoen- coders. Advances in Neural Information Processing Systems, 38:82318–82355, 2026. [23]Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders. arXiv preprint arXiv:2503.17547, 2025. [24]Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, et al. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. transformer circuits thread, 2024. [25]David Chanin and Adrià Garriga-Alonso. Sparse but wrong: Incorrect l0 leads to incorrect features in sparse autoencoders. arXiv preprint arXiv:2508.16560, 2025. [26]Emmanuel J Candès, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on information theory, 52(2):489–509, 2006. [27]Emmanuel J Candes and Terence Tao. Near-optimal signal recovery from random projections: Universal encoding strategies? IEEE transactions on information theory, 52(12):5406–5425, 2006. [28]David L Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4): 1289–1306, 2006. [29] Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996. [30]Michal Aharon, Michael Elad, and Alfred M Bruckstein. On the uniqueness of overcomplete dictionaries, and a practical way to retrieve them. Linear algebra and its applications, 416(1): 48–67, 2006. [31] Christopher J Hillar and Friedrich T Sommer. When can dictionary learning uniquely recover sparse data from subsamples? IEEE Transactions on Information Theory, 61(11):6290–6297, 2015. [32] Sanjeev Arora, Rong Ge, and Ankur Moitra. New algorithms for learning incoherent and overcomplete dictionaries. In Conference on Learning Theory, pages 779–806. PMLR, 2014. 20 [33]Sanjeev Arora, Rong Ge, Tengyu Ma, and Ankur Moitra. Simple, efficient, and neural algorithms for sparse coding. In Conference on learning theory, pages 113–149. PMLR, 2015. [34]Charles J Garfinkle and Christopher J Hillar. On the uniqueness and stability of dictionaries for sparse representation of noisy signals. IEEE Transactions on Signal Processing, 67(23): 5884–5892, 2019. [35]Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv:1312.5663, 2013. [36]Thomas K Landauer and Susan T Dumais. A solution to plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological review, 104(2):211, 1997. [37]Meera Sitharam, Mohamad Tarifi, and Menghan Wang. An incidence geometry approach to dictionary learning. arXiv preprint arXiv:1402.7344, 2014. [38] Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang, Gustavo Hernández Ábrego, Shih-Cheng Huang, Aashi Jain, Daniel Salz, Sonam Goenka, Chaitra Hegde, Ji Ma, et al. Gemini embedding 2: A native multimodal embedding model from gemini. arXiv preprint arXiv:2605.27295, 2026. [39]Yauhen Babakhin, Radek Osmulski, Ronay Ak, Gabriel Moreira, Mengyao Xu, Benedikt Schifferer, Bo Liu, and Even Oldridge. Llama-embed-nemotron-8b: A universal text embedding model for multilingual and cross-lingual tasks. arXiv preprint arXiv:2511.07025, 2025. [40]Mark Muchane, Sean Richardson, Kiho Park, and Victor Veitch. Incorporating hierarchical semantics in sparse autoencoder architectures. arXiv preprint arXiv:2506.01197, 2025. [41]Valérie Costa, Thomas Fel, Ekdeep S Lubana, Bahareh Tolooshams, and Demba Ba. From flat to hierarchical: Extracting sparse representations with matching pursuit. Advances in Neural Information Processing Systems, 38:34381–34424, 2026. [42] Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek. Interpreting clip with hierarchical sparse autoencoders. arXiv preprint arXiv:2502.20578, 2025. [43]Tue M Cao, Hoang X Nhat, Raed Alharbi, Phi Le Nguyen, and My T Thai. Tree sae: Learning hierarchical feature structures in sparse autoencoders. arXiv preprint arXiv:2605.07922, 2026. [44] Josh Engels, Eric Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear. In International Conference on Learning Representations, volume 2025, pages 84591–84622, 2025. [45] Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Jack Merullo, et al. Do sparse autoencoders capture concept manifolds? arXiv preprint arXiv:2604.28119, 2026. [46] Thomas Fel, Matthew Kowal, Mozes Jacobs, Dron Hazra, Usha Bhalla, Lee Sharkey, Lucius Bushnaq, Satchel Grant, Tal Haklay, Thomas Icard, et al. Structuring sparsity: Block-sparse featurizers capture visual concept manifolds. arXiv preprint arXiv:2606.25234, 2026. [47]Thomas Fel, Binxu Wang, Michael A Lepori, Matthew Kowal, Andrew Lee, Randall Balestriero, Sonia Joseph, Ekdeep S Lubana, Talia Konkle, Demba Ba, et al. Into the rabbit hull: From task-relevant concepts in dino to minkowski geometry. arXiv preprint arXiv:2510.08638, 2025. 21 [48]Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024. [49]Florian P Mahner, Johannes Roth, Ka Chun Lam, Michael F Bonner, Francisco Pereira, and Martin N Hebart. Characterizing universal object representations across vision models. arXiv preprint arXiv:2605.13675, 2026. [50]Rishi Jha, Collin Zhang, Vitaly Shmatikov, and John Morris. Harnessing the universal geometry of embeddings. Advances in Neural Information Processing Systems, 38:45963–45987, 2026. [51]Amil Dravid, Yossi Gandelsman, Alexei A Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in a model zoo. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1934–1943. IEEE, 2023. [52]Jon Kleinberg and Manish Raghavan. Algorithmic monoculture and social welfare. Proceedings of the National Academy of Sciences, 118(22):e2018340118, 2021. [53]Kenny Peng and Nikhil Garg. Monoculture in matching markets. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps://openreview.net/ forum?id=P5yezHuMSS. [54] Fan Wu, Emily Black, and Varun Chandrasekaran. Generative monoculture in large language models. In The Thirteenth International Conference on Learning Representations, 2025. [55] Vishakh Padmakumar and He He. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations, 2024. URLhttps: //openreview.net/forum?id=Feiz5HtCD0. [56] Manish Raghavan. Competition and diversity in generative AI. arXiv preprint arXiv:2412.08610, 2024. [57]Johan Ugander and Ziv Epstein. The art of randomness: Sampling and chance in the age of algorithmic reproduction. Harvard Data Science Review, 6(4), 2024. [58] Anil R Doshi and Oliver P Hauser. Generative AI enhances individual creativity but reduces the collective diversity of novel content. Science advances, 10(28):eadn5290, 2024. [59]Shashwat Goel, Joschka Strüber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. Great models think alike and this undermines AI oversight. In Forty-second International Conference on Machine Learning, 2025. [60]Elliot Myunghoon Kim, Avi Garg, Kenny Peng, and Nikhil Garg. Correlated errors in large language models. In Forty-second International Conference on Machine Learning, 2025. [61] Connor Toups, Rishi Bommasani, Kathleen Creel, Sarah Bana, Dan Jurafsky, and Percy S Liang. Ecosystem-level analysis of deployed machine learning reveals homogeneous outcomes. Advances in Neural Information Processing Systems, 36:51178–51201, 2023. [62]Nathanael Jo, Nikhil Garg, and Manish Raghavan. The subjectivity of monoculture. arXiv preprint arXiv:2602.24086, 2026. [63]Nikhil Garg. Econcslib: Ai-assisted lean formalization for economics & computation research. arXiv preprint arXiv:2606.13306, 2026. URLhttps://gargnikhil.com/AppliedModelingLib/. 22 [64]Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687–3697, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1404. URLhttps: //aclanthology.org/D18-1404. [65]James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In NAACL-HLT, 2018. [66]Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=wCu6T5xFjeJ. [67] Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris Callison-Burch. Gooaq: Open question answering with diverse answer types. arXiv preprint, 2021. [68] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259. [69]Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268, 2016. URLhttp: //arxiv.org/abs/1611.09268. [70]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019. doi: 10.1162/tacl_a_00276. URL https://aclanthology.org/Q19-1026/. [71] Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. A full-text learning to rank dataset for medical information retrieval. 2016. URLhttp://w.cl.uni-heidelberg. de/~riezler/publications/papers/ECIR2016.pdf. [72] Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. Paq: 65 million probably-asked questions and what you can do with them. 2021. [73]David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-main.609. URL https://aclanthology.org/2020.emnlp-main.609/. 23 [74]Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1264. URL https://aclanthology.org/D16-1264. [75]Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. triviaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017. [76]Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 2024. [77]Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics, 11:1114–1131, 2023. URL https://aclanthology.org/2023.tacl-1.63/. 24 Appendix contents A Proof of Theorem 1 (Existence of Atomic Features)26 B Proofs of Theorems 2 and 3 (the Recovery Principle)29 C Experimental Setup36 C.1 Data and SAE training details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 C.2 Matching criteria and computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 C.3 P1: Persistence across SAE widths . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 C.4 P2: Shared features across training distributions . . . . . . . . . . . . . . . . . . . . 40 C.5 P3: Hierarchical datasets and feature-recovery protocol . . . . . . . . . . . . . . . . . 41 D Extended Experiments for P1: Persistence across Size43 D.1 Fixed sparsity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 D.2 Variation by seed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 D.3 Exclusive matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 D.4 Persistence by activation similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 D.5 Sensitivity to the matching threshold . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 D.6 Feature activation rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 D.7 Relationship between persistence and prevalence . . . . . . . . . . . . . . . . . . . . 49 E Extended Experiments for P2: Stability across Distributions50 E.1 Cross-distribution stability under exclusive matching . . . . . . . . . . . . . . . . . . 50 E.2 Cross-distribution stability by activation similarity . . . . . . . . . . . . . . . . . . . 51 E.3 Feature prevalence and cross-distribution matching . . . . . . . . . . . . . . . . . . . 52 F Extended Experiments for P3: Hierarchical Feature Recovery53 F.1 Hierarchical recovery at fixed sparsity . . . . . . . . . . . . . . . . . . . . . . . . . . 53 F.2 Counts and proportions of recovered categories and adjacent pairs . . . . . . . . . . . 53 F.3 Recovery of letter-prefix hierarchies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 F.4 Empirical extraction of feature hierarchies . . . . . . . . . . . . . . . . . . . . . . . . 57 G Simulation experiments61 G.1 Recovery under width and sparsity variation . . . . . . . . . . . . . . . . . . . . . . . 61 G.2 Persistent features across simulated SAE widths . . . . . . . . . . . . . . . . . . . . . 64 H A Molecular Model of SAE Scaling67 H.1 Computing the empirical molecules . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 I An Atomic View of Feature Manifolds71 J An Atomic View of the Platonic Representation Hypothesis75 25 A Proof of Theorem 1 (Existence of Atomic Features) For a matrix A∈ R d×M , define U K (A) := [ S⊆[M] |S|=K span(A S ),(17) where A S :=a i i∈S . Lemma 3. We have that U K (A)⊆ U K ′ (B)(18) Proof.FixS ⊆[M] with|S|=K, and writeE S :=spane i :i ∈ S. For eachT ⊆[M ′ ] with |T| = K ′ , define the linear subspace L T :=w ∈ E S : Aw ∈ span(B T ).(19) Since Az(x) = Bz ′ (x) and z ′ (x) is at most K ′ -sparse, z(X)∩ E S ⊆ [ T⊆[M ′ ] |T|=K ′ L T .(20) ByK-richness, these finitely many linear subspaces cannot all be proper subspaces ofE S . Thus L T = E S for some T, which implies span(A S ) = A(E S )⊆ span(B T ).(21) Taking the union over all S ⊆ [M ] with |S| = K gives U K (A)⊆ U K ′ (B), as desired. The inclusion (18) is the primary consequence of richness that we invoke. We first prove thatK ′ ≥ K.Invoking Proposition 3, we have thatU K (A)⊆ U K ′ (B).Consider S ⊆[M] with|S|=K.Thenspan(A S )⊆ U K (A)⊆ U K ′ (B).Since a finite-dimensional subspace cannot be covered by finitely many proper linear subspaces, there exists someT ⊆[M ′ ] with |T|=K ′ such thatspan(A S )⊆ span(B T ).Since|S|=K < spark(A),as implied by the lower RIP, rank(A S ) = K. Therefore, K ′ = rank(B T )≥ K. Next we prove that ifM ′ ,K ′ =M,K, then (A,z)≡(B,u).Suppose thatM ′ =MandK ′ =K. Invoking Proposition 3, we have thatU K (A)⊆ U K ′ (B).As in the proof above, for eachS ⊆[M] with |S| = K, there must exist some T ⊆ [M ] with |T| = K such that span(A S )⊆ span(B T ). Since both areK-dimensional (sincespark(A), spark(B)>2K), we must havespan(A S ) =span(B T ). Defineg: [M] K → [M] K ,S 7→ T.Sincespan(A S )̸=span(A S ′ ) whenS ̸=S ′ (again from the spark condition), g is a bijection. Moreover, |S∩ S ′ | = dim(span(A S )∩ span(A S ′ )) = dim(span(B g(S) )∩ span(B g(S ′ ) )) =|g(S)∩ g(S ′ )|. (22) Now for eachi∈[M], we can chooseS,S ′ ∈ [M] K such thatS∩ S ′ =i,sog(S)∩ g(S ′ ) =jfor some j ∈ [M ]. This implies that span(a i ) = span(A S )∩ span(A S ′ ) = span(B g(S) )∩ span(B g(S ′ ) ) = span(b j ).(23) Sob j =a i since columns are unit norm and coefficients must be non-negative. Since columns ofA are pairwise linearly independent, eachimaps to a distinctjaccording to a permutation matrixP. 26 C N (C) r Figure 4: A sketch of how the tree construction completes the proof. Removing one vertexrfromC, there exists a matching denoted by the solid lines fromC\rtoN(C) (by the “Hall minimality” of C). We can then construct a tree onCwhere the existence of an edgeijimplies thatm(j)∈ F i (so F i ∩ F j is non-empty). Splitting the tree into its two parts, this gives us a subset ofCthat has a large neighborhood, providing the desired contradiction. Then for eachx∈ X,Bz ′ (x) =Az(x).Sincespark(B)>2K,there is a uniqueK-sparse choice of z ′ (x). In particular, z ′ (x) = P T z(x). Finally, we show that if M ′ < M, then K ′ ≥ 2K. The proof rests on the following new characterization of when Hall’s condition holds: if Hall’s condition is satisfied for subsets of size at most 2Kand subsets of size at mostKhave neighborhoods of size at most 2K− 1, then Hall’s condition is satisfied globally. Lemma 4 (Local-to-global Hall lemma). LetG= (X,Y,E) be a bipartite graph, and letK ≥1. For S ⊆ X, let N (S) :=y ∈ Y : (x,y)∈ E for some x∈ S. Suppose that for every S ⊆ X, (i) if |S|≤ 2K, then |N (S)|≥|S|; (i) if |S|≤ K, then |N (S)|≤ 2K− 1. ThenGsatisfies Hall’s condition:|N(S)| ≥ |S|for everyS ⊆ X.Consequently, there exists a matching that saturates X, and in particular |Y|≥|X|. Proof of Proposition 4. Assume for sake of contradiction that (i) and (i) are satisfied, but that there exists C ⊆ X such that |N (C)| <|C|. Take C to be inclusion minimal, in the sense that |N(C ′ )| ≥ |C ′ |for allC ′ ⊊ C.Then|N(C)|=|C|−1,since otherwise we can freely remove any vertex fromCto obtainC ′ violating Hall’s condition. By (i), we have that|C|≥2K+ 1.We now show that (i) is contradicted. Choose anyr ∈ C. By the minimality ofC, every strict subset ofCsatisfies Hall’s condition. Therefore, we can construct a perfect matching m : C\r→ N (C). We now iteratively construct a treeT= (C,E T ) alongside a bijectiong:E T → N(C). Start with the set of verticesR=rand the empty set of edgesE T =∅. At each step, choose a vertex j ∈ C \ Rsuch thatm(j)∈ N(i) for somei ∈ R.Then addjtoR, the edgeijtoE T , and set g(ij) :=m(j).Such a vertexjexists as long asR̸=C. Indeed, suppose that there were not such a vertex. This would mean that for everyu∈ N(R), m −1 (u)∈ R\r. Sincemis a bijection, this implies that |N (R)|≤|R|− 1. Therefore, if R̸= C, this would contradict the minimality of C. 27 A key property of T is that if g(ij) = m(j), then m(j)∈ N (i) and m(j)∈ N (j). Now take any subtree ofTwith 2K+ 1 vertices (equivalently, 2Kedges). Let the two sides of the tree (which is bipartite) have verticesS 1 andS 2 . Without loss of generality, suppose|S 1 |≤ K. Every edgeijin the subtree has one endpoint inS 1 . Therefore, for each such edgeij,g(ij) is in N (S 1 ). Since g is a bijection, this implies that |N (S 1 )|≥ 2K. This contradicts (i), as desired. Finishing the proof. We now complete the proof. Invoking Proposition 3, we have thatU K (A)⊆ U K ′ (B).Therefore, for alli∈[M], sincea i ∈ U K (A)⊆ U K ′ (B), a i hasK ′ -sparse representation in B. Sincespark(B)>2K ′ ,this representation is unique. LetF i ⊆[M ′ ] denote the support of this representation. Now consider a bipartite graphG= ([M],[M ′ ],E) whereE=(i,j) :i∈[M],j ∈ F i .Then we have N (S) =∪ i∈S F i ⊆ [M ′ ]. Our approach is to show that ifK ′ ≤2K−1, thenGsatisfies the two conditions in Proposition 4, implying that M ′ ≥ M. To show (i) holds, observe that since|S| ≤2K < spark(A), thenrank(A S ) =|S|. Also, span(A S )⊆ span(B ∪ i∈S F i ).Now observe that|N(S)|=|∪ i∈S F i |≥ rank(B ∪ i∈S F i )≥ rank(A S ) =|S|. (Notice that this property holds regardless of K ′ .) To show (i) holds, observe that if|S| ≤ K,span(A S )⊆ U K (A)⊆ U K ′ (B).Since a subspace can’t be covered by the union of a finite number of strictly lower-dimensional subspaces, there exists T S ⊆[M ′ ] with|T S |=K ′ such thatspan(A S )⊆ span(B T S ).For eachi ∈ S, sincea i ∈ span(A S ) andF i is the support of the uniqueK ′ -sparse representation ofa i inB, thenF i ⊆ T S .Therefore, |N (S)| =|∪ i∈S F i |≤|T S |≤ K ′ ≤ 2K− 1, as desired. Define a featurization of f to be a dictionary that exactly represents f. Theorem 4 (Generic rigidity). Fixd,M,KwithM > d≥2. For almost everyA= (a 1 ,...,a M )∈ (S d−1 ) M , the following holds. Suppose that (A,z) is aK-rich (M,K)-featurization off:X → R d . Then every (M ′ ,K ′ )-featurization (A ′ ,z ′ ) of f, for M ′ > d, satisfies K ′ ≥ 1 + (d− 1) M − M ′ M .(24) Since d + 1≥ spark(A ′ ) > 2K ′ , (24) implies that M ′ > M 2 . Proof of Theorem 4. Suppose that (A,z) is aK-rich (M,K)-featurization off:X → R d . Then by Proposition 3, we have that a i ∈ U K ′ (B). We show that if K ′ < 1 + (d− 1) M − M ′ M ,(25) then for almost everyA= (a 1 ,a 2 ,· ,a M )∈(S d−1 ) M , there does not existB ∈ R d×M ′ such that a 1 ,a 2 ,· ,a M all haveK ′ -sparse representations inB. (Notice that this argument does not require thatU K ′ (B) include all ofU K (A),but rather justa 1 ,a 2 ,· ,a M , so it seems natural that the bound given here could be further improved.) The idea is a dimension counting argument on parameter spaces. In particular, we first define a smooth manifold that parameterizesA. We then define a smooth manifold that parameterizes (M ′ ,K ′ )-featurizationsB,z ′ that can representa 1 ,a 2 ,· ,a M . Finally, we define a smooth map be- tween theB-manifold and theA-manifold, with the property that every representableA-featurization must be in the image of this map. Then invoking Sard’s theorem, if the dimension of theB-manifold is smaller than theA-manifold, then the image of the map has measure zero. So for a measure one subset of A, no (M,K)-featurization A,z has an (M ′ ,K ′ )-featurization. 28 B ∗ M ′ ,K ′ ,T Θ◦ Φ (Θ◦ Φ)(B M ′ ,K ′ ,T ) A M Figure 5: Whendim(B M ′ ,K ′ ,S )< dim(A M ),the set of (M,K)-factorizations that also have an (M ′ ,K ′ )-factorization is measure zero under a generic distribution over (M,K)-factorizations. LetA M = (S d−1 ) M . LetB M ′ ,K ′ ,T = (S d−1 ) M ′ × Q M i=1 S |T i |−1 . LetT= (T 1 ,· ,T M ) where T i ⊆ [M ′ ] with 1≤|T i |≤ K ′ . Define the smooth function Φ :B M ′ ,K ′ ,T → (R d ) M such that (B,γ 1 ,γ 2 ,· ,γ M )7→ X j∈T 1 γ 1j b j , X j∈T 2 γ 2j b j ,· , X j∈T M γ Mj b j .(26) Now define Θ : (R d \0) M →A M : (a ′ 1 ,a ′ 2 ,· ,a ′ M )7→( a ′ 1 ∥a ′ 1 ∥ , a ′ 2 ∥a ′ 2 ∥ ,· , a ′ M ∥a ′ M ∥ ),which is also smooth. LetB ∗ M ′ ,K ′ ,T = Φ −1 ((R d \0) M ). Since (R d \0) M is open,B ∗ M ′ ,K ′ ,T is itself a smooth manifold. B ∗ M ′ ,K ′ ,T has dimension at most M ′ (d− 1) + M (K ′ − 1). A M has dimension M (d− 1). So if M ′ (d− 1) + M (K ′ − 1) < M (d− 1),(27) by Sard’s theorem, (Θ◦Φ)(B ∗ M ′ ,K ′ ,T )⊆A M has measure zero. Now observe that every (a 1 ,a 2 ,· ,a M ) that has aK ′ -sparse decomposition in (b 1 ,b 2 ,· ,b M ′ ) must lie in the image of (Θ◦Φ)(B ∗ M ′ ,K ′ ,T ) for someT. Since there are a finite number of supportsT, and the union of a finite number of measure zero sets has measure zero, this implies that with probability one, (a 1 ,a 2 ,· ,a M ) does not have aK ′ -sparse decomposition in any (b 1 ,b 2 ,· ,b M ′ ). Similarly, we can repeat the argument for all M ′ ,K ′ satisfying (27). Finally, observe that since d + 1≥ spark(A ′ ) > 2K ′ , then K ′ − 1 < d−1 2 , so d− 1 2 > (d− 1) M − M ′ M (28) M 2 > M − M ′ (29) M ′ > M 2 ,(30) as desired. B Proofs of Theorems 2 and 3 (the Recovery Principle) Let us recall our assumptions. We assume throughout thatAand ˆ Ahave unit-norm columns and satisfy the (γ,2K)-lower restricted isometry property. To simplify, we simply say that a matrix is (γ,2K)-stable. We also assume thatz(x) andˆz(x) are nonnegative, and thatˆz(x) is at most K-sparse. The atomic supportS(x) :=supp(z(x)) has sizeKalmost surely. Conditional on any 29 positive-probability supportS, the coefficientsz S (x) lie in (0,1) K and have densityh S satisfying c − ≤ h S ≤ c + . The densities may differ across supports. All probabilities are with respect toD. We abbreviate ˆ A (D,m) , ˆz (D,m) by ˆ A, ˆz and write E := L D ( ˆ A, ˆz), E S := E x∼D h ∥Az(x)− ˆ Aˆz(x)∥ 2 2 S(x) = S i . For equal-dimensional subspacesU,V ⊆ R d , letgap(U,V) :=sup u∈U:∥u∥ 2 =1 dist(u,V); the same value is obtained after exchangingU,V. We setgap(0,0) = 0. The first three lemmas do not require regularity, so we can use them for both the ordinary and hierarchical settings. The core idea is the following lemma, which shows that if a full atomic support span is accurately reconstructed by the learned dictionary, then there must be a single learned support whose span matches the atomic span (in a way that is independent ofmandM). Then, by intersecting these spans, we can show the recovery of individual atoms. A key challenge in showing the result is that the atomic span can, in theory, be “tiled” by many different learned spans (the number of which depends on m). Lemma 5. For everyε >0, there existsδ=δ(ε,K,γ,c − )>0 such that any positive-probability supportSwithE S ≤ δhas a learned supportT ⊆[m] of sizeKsatisfyinggap(span(A S ), span( ˆ A T ))≤ ε. Proof. Fixε >0. Setr 0 :=γ/4 andr j+1 :=γr j /(16K) forj= 0,...,K −1. WriteR:=r K and choose α := R minε, 1/8. Note that these quantities are independent of m,M. DefineU K ( ˆ A) := S T⊆[m]:|T|≤K span ( ˆ A T ) andg(z) :=dist(A S z,U K ( ˆ A)). The conditional density lower bound givesE S ≥ c − R (0,1) K g (z) 2 dz. Moreover, since distance to a fixed set is 1-Lipschitz and ∥A S ∥ op ≤ √ K, we have |g(z)− g(z ′ )|≤ √ K∥z− z ′ ∥ 2 . Supposeg(z 0 )> αat somez 0 ∈[1/4,3/4] K . Theng(z)> α/2 throughout the coefficient ball of radiusα/(2 √ K) centered atz 0 . This ball lies inside (0,1) K since unit norms and stability imply γ ≤1, and our choice ofαmakes its radius smaller than 1/4. Writingω K for the volume of the unit ball in R K , choose δ := c − ω K 2 K+3 K K/2 α K+2 . The contribution of this ball would giveE S ≥ c − (α 2 /4)ω K (α/(2 √ K)) K = 2δ, a contradiction. Thus g(z)≤ α throughout [1/4, 3/4] K . WriteV:=span(A S ) andy 0 :=A S (1/2,...,1/2). Sinceσ min (A S )≥ γ, the imageA S [1/4,3/4] K contains the ballB 0 :=B V (y 0 ,r 0 ). Indeed, any displacement inVof norm at mostγ/4 equalsA S w for some ∥w∥ 2 ≤ 1/4. Here and below, B V denotes a closed ball taken within V . For eachy ∈ B 0 , choose a best signedK-sparse coefficient vectorv(y). Such a vector exists becauseU K ( ˆ A) is a finite union of subspaces. The preceding bound gives∥y− ˆ Av(y)∥ 2 ≤ α. Since v(y)− v(y ′ ) is at most 2K-sparse, stability and the triangle inequality imply that for all y,y ′ ∈B 0 , ∥v(y)− v(y ′ )∥ 2 ≤ ∥ ˆ A(v(y)− v(y ′ ))∥ 2 γ ≤ ∥y− y ′ ∥ 2 + 2α γ (31) We now construct nested ballsB j :=B V (y j ,r j ) and setsT j ⊆[m] of sizejsuch thatT j ⊆ supp(v(y)) throughout B j . Initially, take T 0 = ∅. SupposeB j ,T j have been constructed, withj < K. SincedimV=K > j, choose a unit vector inVorthogonal tospan( ˆ A T j ). At least one of the two points obtained by displacingy j by±r j /2 in 30 this direction has distance at leastr j /2 fromspan( ˆ A T j ). Call this pointy j+1 . Then, since errors are at most α and since columns are unit-norm, r j 2 ≤ dist y j+1 , span( ˆ A T j ) ≤ α + X ℓ/∈T j |v ℓ (y j+1 )|.(32) Sinceα≤ r j /4 andv(y j+1 ) has at mostKnonzero entries, somei /∈ T j satisfies|v i (y j+1 )|≥ r j /(4K). SetT j+1 :=T j ∪iandB j+1 :=B V (y j+1 ,r j+1 ). This ball lies insideB j , since its center is displaced byr j /2 andr j+1 ≤ r j /2. For everyy ∈B j+1 ,(31)and 2α≤ r j+1 give|v i (y)− v i (y j+1 )|≤ 2r j+1 /γ=r j /(8K). Hence|v i (y)|≥ r j /(8K)>0 throughout the smaller ball. The coordinates inT j remain nonzero by the induction hypothesis, completing the construction. AfterKsteps,T:=T K has sizeK, so sparsity forcessupp(v(y)) =TthroughoutB K . Conse- quently, dist(y, span( ˆ A T ))≤ α for every y ∈ B V (y K ,R). Finally, letP T be the orthogonal projection ontospan( ˆ A T ). For any unit vectorw ∈ V, both y K + Rw and y K − Rw belong to the final ball. Therefore, 2R∥(I − P T )w∥ 2 ≤∥(I − P T )(y K + Rw)∥ 2 +∥(I − P T )(y K − Rw)∥ 2 ≤ 2α. Noting thatIis the projection ontospan(A S ) and that bothA S and ˆ A T have dimensionK, so taking the supremum over unit vectors w ∈ V bounds their gap by α/R≤ ε. The next lemma shows that approximate matching of support spans preserves their intersections. Importantly, its bound does not depend on the number of supports being intersected. Lemma 6. Suppose a nonempty finite collection of pairs S ℓ ,T ℓ , with |S ℓ | =|T ℓ | = K, satisfies gap(span(A S ℓ ), span( ˆ A T ℓ ))≤ rfor every ℓ. Write I := T ℓ S ℓ and J := T ℓ T ℓ . If r(1 + 2K/γ) < 1, then |I| =|J|,gap span(A I ), span( ˆ A J ) ≤ r(1 + 2K/γ). Proof. Take a unit vectoru ∈ span(A I ). For eachℓ, choosev ℓ ∈ R m , supported onT ℓ , such that ∥u− ˆ Av ℓ ∥ 2 ≤ r. Fix one pair, indexed by 1. For everyj ∈ T 1 \ J, someT ℓ omitsj. Stability then gives|(v 1 ) j |≤∥v 1 − v ℓ ∥ 2 ≤ 2r/γ. Deleting these at mostKcoordinates fromv 1 , and using the unit norms of the learned columns, gives dist(u, span( ˆ A J ))≤∥u− ˆ Av 1 ∥ 2 + X j∈T 1 |(v 1 ) j |≤ r + 2Kr/γ. ExchangingA,S ℓ with ˆ A,T ℓ gives the same bound in the reverse direction. If the two intersection spans had different dimensions, the larger would contain a unit vector orthogonal to the smaller, contradicting the bound being less than 1. Their dimensions are|I|and|J|by stability, proving both claims. This argument also covers the case in which an intersection is empty. We next connect these geometric matches with the actual learned activations. This step uses both nonnegativity ofˆzand the upper density bound. It does not assume thatˆz(x) minimizes the reconstruction error for each x. 31 Lemma 7 (Activation recovery). For everyη ∈(0,1), there existr,c,τ >0, depending only on η,K,γ,c + , with the following property. LetGbe a collection of positive-probability atomic supports. Suppose eachS ∈Ghas a learned supportT S of sizeKsatisfyinggap(span(A S ), span( ˆ A T S ))≤ r. Suppose also that an atomic coordinate iof prevalencep i :=Pr(z i >0)>0 and a learned coordinatejsatisfyj ∈ T S if and only ifi ∈ S, for every S ∈G. If E + Pr(S(x) /∈G)≤ cp i , then F1 D 1z i > 0,1ˆz j > τ > η. Proof.We first derive a bound for any 0≤ r ≤1/2 andτ >0 satisfying 2Kr ≤ γτ. Write β := Pr(S(x) /∈G). FixS ∈ G, and letP T S denote orthogonal projection ontospan( ˆ A T S ). LetH S be theK × K matrix satisfying ˆ A T S H S =P T S A S . It exists and is unique because ˆ A T S has full column rank. For an input with supportS, letv S (x)∈ R m have coordinatesH S z S (x) onT S and zero elsewhere. These auxiliary coefficients may have either sign. Since the atomic coefficients lie in (0,1) K and the columns have unit norm,∥Az(x)∥ 2 ≤ K. The gap bound therefore gives∥Az(x)− ˆ Av S (x)∥ 2 ≤ Kr. Writinge(x) :=∥Az(x)− ˆ Aˆz(x)∥ 2 , stability and the triangle inequality imply ∥ˆz(x)− v S (x)∥ 2 ≤ ∥ ˆ A(ˆz(x)− v S (x))∥ 2 γ ≤ e(x) + Kr γ .(33) Here stability applies because ˆz(x)− v S (x) is at most 2K-sparse. Ifj ∈ T S , the coordinate (v S (x)) j is a linear function ofz S (x) with at least one coefficient of magnitude γ/(2K) or larger. To see this, for every w ∈ R K , √ K∥H S w∥ 2 ≥∥P T S A S w∥ 2 ≥ p 1− r 2 ∥A S w∥ 2 ≥ γ 2 ∥w∥ 2 . Thusσ min (H S )≥ γ/(2 √ K). SinceH S is square, each row has at least this norm, and hence has an entry of magnitude at least γ/(2K). Holding the otherK−1 atomic coefficients fixed, the values of the corresponding coefficient for which|(v S (x)) j | ≤ 2τform an interval of length at most 8Kτ/γ. Integrating over the remaining coordinates and using h S ≤ c + gives Pr |(v S (x)) j |≤ 2τ S(x) = S ≤ 8Kc + τ γ .(34) No independence of the coefficients is needed. By Markov’s inequality, outside an event of probability at mostβ+ 4E/(γ 2 τ 2 ), we haveS(x)∈G ande(x)≤ γτ/2. On this complement,(33)andKr ≤ γτ/2 give∥ˆz(x)− v S (x)∥ 2 ≤ τ, where S = S(x). Ifi /∈ S, thenj /∈ T S , so (v S (x)) j = 0 andˆz j (x)≤ τ. Thus there is no false positive on this complement. Ifi∈ Sbutˆz j (x)≤ τ, nonnegativity ofˆz j (x) implies−τ ≤(v S (x)) j ≤2τ. Summing (34)over supports inGcontainingi, the probability of these false negatives is at most (8Kc + τ/γ)p i . LetY:= 1z i >0and b Y := 1ˆz j > τ. The total disagreement probability is therefore at most β + 4E/(γ 2 τ 2 ) + (8Kc + τ/γ)p i . Since Pr(Y = 1) = p i > 0, the definition of the F1 score gives F1 D (Y, b Y ) = 1− Pr D (Y ̸= b Y ) Pr D (Y = 1) + Pr D ( b Y = 1) ≥ 1− β p i − 4E γ 2 τ 2 p i − 8Kc + τ γ . (35) 32 Finally, chooseτ >0 sufficiently small that 8Kc + τ/γ <(1− η)/2, and choose 0< r ≤ min1/2,γτ/(2K). Choosec >0 sufficiently small thatc(1 + 4/(γ 2 τ 2 ))<(1− η)/2. These choices depend only onη,K,γ,c + . IfE+β ≤ cp i , then the sum of the first two error terms in(35)is at most c(1 + 4/(γ 2 τ 2 )), so the F1 score exceeds η. The ordinary recovery principle. We now impose regularity of the support law: for a fixed ρ∈(0,1] and distinct atomsi,kwithp i >0, assumePr(k /∈ S(x)| i∈ S(x))≥ ρ. Letσorder the atom prevalences nonincreasingly and letT(m) := P M j=m+1 p σ(j) . WriteL ∗ D (m) for the infimum of the dictionary learning loss over the unit-norm, (γ,2K)-stable reference class with nonnegative, at most K-sparse codes. We do not need the infimum to be attained. Proof of Theorem 2.Retain themmost prevalent columns ofA, together with their true coefficients. This gives an admissible dictionary. For every input, the squared norm of the omitted sum is at most K P k/∈σ([m]) z k (x) 2 , which is at most K times the number of omitted active atoms. Therefore, E ≤ C 0 L ∗ D (m)≤ C 0 KT (m).(36) We will show that an atom of prevalencep i >0 is recovered wheneverE ≤ cp i , for a sufficiently small constant c > 0. It suffices to take 0< η <1. Letr ∗ ,c ∗ ,τbe the constants supplied by Lemma 7. Choose 0< r ≤ r ∗ small enough thatr(1 + 2K/γ)< minγ/(4 √ 2), p 1− η 2 . Letδbe the threshold supplied by Lemma 5 withε=r, decreasing it if necessary so thatδ < c − γ 2 (γ/(8K)) K−1 /32. Finally, choosec >0 such thatc < ρδandc(1 +δ −1 )≤ c ∗ . These choices depend only on the parameters in the theorem. Call a positive-probability supportSgood ifE S ≤ δ, and writeGfor the good supports. Their complement has probabilityβ ≤ E/δ. For each good support, chooseT S as in Lemma 5, so its span has gap at most r from span(A S ). Fix an atomiwithp i >0 andE ≤ cp i , and letG i :=S ∈G:i∈ S. Sinceβ < ρp i ≤ p i , this collection is nonempty. For everyk ̸=i, regularity givesPr(i∈ S(x),k /∈ S(x))≥ ρp i > β. Hence some good support containsiand omitsk, implying T S∈G i S =i. Lemma 6 now gives a single learned coordinate j such that \ S∈G i T S =j,gap span(a i ), span(ˆa j ) ≤ r(1 + 2K/γ).(37) Thusj ∈ T S whenever a good support containsi. If a good supportSomitsi, addingSto the preceding intersection makes the atomic intersection empty. Lemma 6 makes the learned intersection empty as well, soj /∈ T S . We have provedj ∈ T S if and only ifi∈ Son all good supports, not just on those containing i. The line bound in(37)gives|⟨a i , ˆa j ⟩| > η. We must also fix its sign. If the inner product were negative, the same bound and our choice ofrwould give∥a i +ˆa j ∥ 2 < γ/4. Choose anyS ∈G i and consider coefficients withz i ∈(1/2,1) andz k ∈(0,γ/(8K)) fork ∈ S\i. Sinceˆzis nonnegative and at mostK-sparse,ˆz+z i e j is at most 2K-sparse and has norm at leastz i . Stability and the triangle inequality therefore give, on this coefficient region, ∥Az− ˆ Aˆz∥ 2 ≥ (γ−∥a i + ˆa j ∥ 2 )z i − X k∈S\i z k ≥ γ/4. The density lower bound gives this region conditional probability at least (c − /2)(γ/(8K)) K−1 . Thus E S ≥ c − γ 2 (γ/(8K)) K−1 / 32, contradicting goodness and our choice ofδ. Consequently,⟨a i , ˆa j ⟩ > η, as required for signed direction recovery. 33 For activation recovery, observe thatE+β ≤ E(1 +δ −1 )≤ c ∗ p i . Lemma 7 therefore applies to the same coordinatejand gives an F1 score greater thanηat thresholdτ. This threshold is shared by all recovered coordinates. Finally, takeC:=C 0 K/c. By(36), the conditionp i ≥ CT(m) implies E ≤ cp i and completes the proof. The hierarchical recovery principle. We make precise the hierarchical regularity invoked in the main text. LetM= 3NandK= 2L, with familiesi,i 1 ,i 2 fori∈[N]. An input chooses a set F(x)⊆[N] of exactlyLactive families. Each active family contributes its parent and exactly one child, soS(x) = S i∈F(x) i,i b i (x) withb i (x)∈1,2. The law ofF(x) satisfiesPr(k /∈ F(x)| i∈ F(x))≥ ρ for distinct parents i,k whenever Pr(i∈ F (x)) > 0. In addition, for a fixed κ∈ (0, 1/2], we require Pr b i (x) = b| F (x) = F ≥ κ(i∈ F, b∈1, 2)(38) for every positive-probability family setF. Conditional on each resulting atomic support, allK coefficients have a joint density betweenc − andc + on (0,1) K , as above. We call such a distribution hierarchical regular. The child choices need not be independent, and neither do the active coefficients. Independent uniform child choices give the special caseκ= 1/2. The child-choice bound and the full conditional density bounds are part of this definition; they do not follow from the child-implies-parent condition alone. As in the main text, now letp i :=Pr(i ∈ F(x)) be the parent prevalences, letσsort these prevalences, and letT(n) := P N j=n+1 p σ(j) . For a learned dictionary of widthm, we setn:=⌊m/3⌋, with T (0) := P N i=1 p i = L. Proof of Theorem 3. Retain all three atoms in each of then=⌊m/3⌋most prevalent families. If 3n < m, fill the remaining slots with unused columns ofAand set their codes to zero. This gives an admissible dictionary of exactlymunit-norm columns. Each omitted active family contributes two omitted atoms, so the same truncation calculation as before gives E ≤ C 0 L ∗ D (m)≤ 2C 0 KT (n).(39) We show that a family of prevalencep i >0 is recovered wheneverE ≤ cp i , for a sufficiently small constant c > 0. Again it suffices to take 0< η <1. Letr ∗ ,c ∗ ,τbe the constants from Lemma 7. Choose 0< r ≤ r ∗ withr(1 + 2K/γ)<1, and letδbe the threshold from Lemma 5 withε=r. Choosec >0 so thatc < κρδandc(1 +δ −1 )≤ κc ∗ . As before, letGcontain the positive-probability supports with E S ≤ δ, choose their learned supportsT S , and writeβ ≤ E/δfor the probability of the remaining supports. Fix a familyiwithp i >0 andE ≤ cp i . Each child has prevalence at leastκp i . Moreover, for a different family k, (38) and parent regularity give Pr i b ∈ S(x), k /∈ F (x) ≥ κ Pr i∈ F (x), k /∈ F (x) ≥ κρp i > β. Thus, even after discarding the bad supports, either child can occur withi, and each chosen child can occur without any specified other family. It follows that the collections below are nonempty and \ S∈G:i∈S S =i, \ S∈G:i b ∈S S =i,i b (b∈1, 2).(40) Indeed, the other child is excluded wheneveri b occurs, and the displayed probability bound excludes every atom belonging to any other family. 34 Applying Lemma 6 to these intersections gives learned sets J i := \ S∈G:i∈S T S =j i , J i b := \ S∈G:i b ∈S T S =j i ,j i b . HereJ i ⊆ J i b because every support containingi b also containsi. The good supports containingiare exactly the union of those containingi 1 and those containingi 2 . HenceJ i 1 ∩ J i 2 =J i , soj i ,j i 1 ,j i 2 are distinct. We claim that each of these learned coordinates has the same membership pattern as its corre- sponding atom across all good supports. For the parent, this is the same argument as in Theorem 2: adding a good support omittingito its intersection forces that support’s learned set to omitj i . For a childi b , every good support containingi b containsJ i b on the learned side. If a good support omits the family, its atomic intersection withi,i b is empty, so its learned intersection withJ i b is empty as well. If it instead containsiand the other child, that atomic intersection isi. Lemma 6 makes the learned intersection a singleton, which must bej i . In particular it omitsj i b . This proves the claimed membership patterns. Finally,E+β ≤ E(1 +δ −1 )≤ κc ∗ p i , and each of the three target coordinates has prevalence at leastκp i . Lemma 7 therefore gives an F1 score greater thanηfor each target, using the distinct learned coordinatesj i ,j i 1 ,j i 2 . The same thresholdτworks for all three and for all recovered families. Take C := 2C 0 K/c and invoke (39). This proves the theorem. 35 C Experimental Setup This section describes the shared data, training setup, and matching metrics, followed by the setup for the main P1, P2, and P3 experiments in Figures 2a to 2c, respectively. Extended experiments (including results and setup) are discussed in Sections D to F. Code. Code to reproduce data preparation, SAE training, and evaluation are given in the repository at https://github.com/kennylpeng/atomic_theory. C.1 Data and SAE training details Training corpus. The full training mix,full, combines 13 source datasets used to fine-tune the Nemotron embedding model. Table 3 counts the stored embedding vectors available to SAE training: a question and its answer, for example, are embedded separately and contribute two vectors. Table 3: SAE training datasets. Source collection# embeddings Emotion [64]19,930 FEVER [65, 66]11,079,420 GooAQ [67]6,024,992 HotpotQA [68]10,564,510 MS MARCO [69]9,351,785 Natural Questions [70]200,462 NFCorpus [71]10,503 PAQ, sampled 10M pairs [72]20,000,000 SciFact [73]11,475 SQuAD [74]175,198 TriviaQA [75]146,692 WebInstructSub [76]4,670,440 MIRACL27,572,151 Full mix (full)89,827,558 Corpus. Each dataset gives a query and an answer. We embed both separately. We sample 10M question-answer pairs from PAQ. For MIRACL [77], we sample up to 500,000 passages per language. All available passages are used when a language has fewer than 500,000. (Due to an error, our nemotron embeddings are missing 500,000 Thai passages and 100,000 sampled Telugu passages present for Gemini. Thus the full corpora contain 89.8M Gemini vectors and 89.2M Nemotron vectors.) When comparing activations between the two embedding families, we use the 89.2M shared embeddings. Embeddings. For gemini embeddings, we usegemini-embedding-2-previewwith the task type “RETRIEVAL_DOCUMENT” and dimensiond= 3072. The nemotron representations are generated fromnvidia/llama-embed-nemotron-8bwithd= 4096. We truncate at 10,000 characters for gemini and 256 tokens for Nemotron. Additional data splits for P2 experiments. We train a separatem= 16,384,k= 64 SAE for each of the following mixtures and each embedding model: 36 • rand1: emotion, fever, gooaq, natural-questions, sampled PAQ, and scifact; 37,336,279 vectors. • rand2: hotpotqa, msmarco, nfcorpus, squad, trivia-qa, and WebInstructSub; 24,919,128 vectors. • wiki: fever, hotpotqa, natural-questions, sampled PAQ, squad, and trivia-qa; 42,166,282 vectors. • no_wiki: emotion, gooaq, msmarco, nfcorpus, scifact, WebInstructSub; 20,089,125 vectors. Each mixture contains six complete prepared datasets. Thewiki/no_wikiandrand1/rand2pairs are two partitions of the same 12 non-MIRACL datasets; both MIRACL collections occur only infull. wiki/no_wikisplits based on if the dataset includes Wikipedia data or not. SAE architecture and objective. We use the ordinary (non-batch) TopK SAE implementation fromhttps://github.com/bartbussmann/BatchTopK/tree/main. For an embeddingx ∈ R d the forward pass is h(x) = ReLU(W enc (x− b dec )),(41) z(x) = TopK k (h(x)),(42) ˆx = W dec z(x) + b dec ,(43) whereW enc ∈ R m×d ,W dec ∈ R d×m , andb dec ∈ R d . There is no encoder bias.TopK k retains thek largest nonnegative activations in each example. LetDbe the set of dead neurons: neurons with no positive post-TopK activation in the last 512 consecutive training batches. For each example,z AuxK (x) retains the largestmin32,|D|entries of the pre-TopK activation vectorh(x) restricted toD, with all other entries set to zero. For a batch of B embeddings, the implemented loss is L = 1 Bd B X i=1 h ∥x i − ˆx i ∥ 2 2 + w aux W dec z AuxK (x i )− (x i − ˆx i ) 2 2 i .(44) The auxiliary term is zero whenDis empty. Its residualx i − ˆx i remains in the gradient computation. We use noℓ 1 penalty. The auxiliary coefficient isw aux = 0.25 except for the Gemini width-131,072 SAE, which usesw aux = 1.0 (we found that at 0.25 in this setting, the number of dead neurons would remain large). Optimization and training budget. We train SAEs given in Table 4. The four additional distribution mixtures give 26 SAEs in the main width/split experiments. We use float32 parameters, batch sizeB= 2048, and Adam with learning rate 5×10 −4 , (β 1 ,β 2 ) = (0.9,0.99),ε= 10 −8 , and no weight decay or learning-rate schedule. Gradients are clipped to global norm 1. The encoder is initialized with Kaiming-uniform weights, the decoder with the normalized transpose of the encoder, and the bias with zeros. Before each optimizer step, decoder directions are normalized and their radial gradient components are removed. All of these SAEs train for 1,000,000 optimizer steps except the Gemini width-131,072 SAE, which trains for 2,000,000 steps. These budgets correspond to 2.048 billion and 4.096 billion sampled embeddings, respectively, with repeated sampling from the finite corpora above. A shard is selected with probability proportional to its row count and supplies four batches before the next shard selection. Each batch is a randomly positioned contiguous block with its rows shuffled; shards smaller than one batch are sampled with replacement. Training uses NVIDIA H100, A100 (80GB), or A6000 GPUs. Seeds are fixed separately for each run, rather than paired between embedding models. The synthetic experiments use a separate training recipe described in Section G. 37 Table 4: Sparsities for the 26 SAEs in the main width and distribution experiments. Subset runs all usem= 16,384 andk= 64. The Gemini width-131,072 run uses 2M steps and auxiliary coefficient 1.0; every other listed run uses 1M steps and coefficient 0.25. Width or subset k 51232 1,02432 2,04832 4,09632 8,19264 16,38464 32,76864 65,536128 131,072128 wiki64 no_wiki64 rand164 rand264 Further sparsity and seed SAE controls. Table 5 lists six additional fixed-k= 128 SAEs and three alternate-seed Gemini SAEs. Table 5: The nine additional SAEs used for the fixed-sparsity and alternate-seed controls. All use 1,000,000 optimizer steps, batch size 2048, and auxiliary coefficient 0.25. ControlModelWidth k Fixed kGemini512 128 Fixed kGemini4,096 128 Fixed kGemini32,768 128 Fixed kNemotron512 128 Fixed kNemotron 4,096 128 Fixed kNemotron 32,768 128 Alternate seed Gemini512 32 Alternate seed Gemini4,096 32 Alternate seed Gemini32,768 64 C.2 Matching criteria and computation The P1, P2, and cross-model comparisons combine a pairwise similarity score with a rule for selecting matches. We define the shared metrics here; the experiment-specific sections specify the dictionaries and evaluation corpora. We use a defualt threshold of t = 0.7 [20]. C.2.1 Direction similarity For nonzero dictionary directionsuandvin the same embedding space, we use signed cosine similarity, s dir (u,v) = ⟨u,v⟩ ∥u∥ 2 ∥v∥ 2 . 38 For SAEs,uandvare decoder columns. The PCA baseline uses|s dir (u,v)|for both maximum- cardinality and exclusive matching, because the signs of principal components are arbitrary. C.2.2 Activation similarity For two SAE features evaluated on the same examplesx 1 ,...,x N , leta n andb n be their continuous post-TopK activations. With ̄a=N −1 P n a n and ̄ b=N −1 P n b n , their signed Pearson correlation is s act (a,b) = P N n=1 (a n − ̄a)(b n − ̄ b) q P N n=1 (a n − ̄a) 2 q P N n=1 (b n − ̄ b) 2 . Every evaluation example receives equal weight, including examples with zero activation. The metric depends on the choice of evaluation data. C.2.3 One-to-one matching and persistence For either similarity score, eligible pairs have similarity at leasttand form the edges of a bipartite graph. A maximum-cardinality matching is the largest set of eligible pairs in which each feature occurs at most once. We denote this maximum cardinality byμ t (U,V). For PCA directions, we use the absolute-cosine modification above. Reported proportions divide the matching count by the source dictionary width, including inactive and unmatched features in the denominator. For all maximum-matching persistence results, we compute a maximum-cardinality matchingM ℓ independently on the threshold graph between the full source dictionaryUand each larger target dictionaryV ℓ , using retained candidate edges for empirical activation comparisons (Section C.2.4). Let S ℓ =u∈ U : there exists v ∈ V ℓ with (u,v)∈ M ℓ , S persistent = s \ ℓ=1 S ℓ . Maximum-cardinality matchings can be nonunique, so the persistent set and its size can depend on tie-breaking. We retain ascending original source and target feature IDs, canonicalize each graph as a sorted CSR matrix, and use SciPy 1.15.3’smaximum_bipartite_matchingwithperm_type=’column’. This fixes the matching choices for these results. C.2.4 Candidate graph construction Direction comparisons evaluate all pairs. For activation comparisons, we use 512-dimensional CountSketch representations to propose 32 candidates per feature in each search direction, with the same evaluation rows and sketch hashes for both dictionaries. We recompute candidate correlations over the full evaluation corpus and retain the five highest-scoring candidates per feature in each direction. Thresholded matching uses the union of retained pairs with correlation at least t. The sketch hash is the fixed SplitMix64 transformation of the original global row index, imple- mented by row_hash in scripts/match_cross_model_sae_activations_131k.py. The within-family width sweeps use 89,827,558 Gemini rows and 89,227,558 Nemotron rows, respectively. Same-width cross-model comparisons use the 89,227,558 aligned shared rows, with hashes based on their original Gemini offsets. Split comparisons instead use the row distribu- tion identified in each evaluation manifest. These frozen manifests record ordered shard iden- tities, offsets, model-checkpoint SHA-256 hashes, and producing-code hashes. The cross-model alignment_audit.jsonadditionally records SHA-256 hashes of each shard’s ordered source-row IDs after checking dataset/configuration/split/shard identity, text columns, and row order across the two 39 models. The reproduction workflows preserve these manifests and retained neighbor arrays; the code archive’s SOURCE_MANIFEST.json identifies the supplied source revision. Pairwise maximum matching is exact on these retained graphs, and its cardinality is a lower bound on exhaustive pairwise Pearson matching. C.2.5 Alternative exclusive matching criterion The exclusive direction comparisons in Figures 6 and 13 count a pair (u,v) only when its similarity exceedstand neither endpoint has another above-threshold partner. We uses(u,v) =s dir (u,v) for SAEs ands(u,v) =|s dir (u,v)|for PCA, applying the same score to pair admission and competing partners. The requirements are s(u,v) > t, s(u,v ′ )≤ t for all v ′ ̸= v, s(u ′ ,v)≤ t for all u ′ ̸= u.(45) The exclusive pairs themselves form a feasible one-to-one matching, so their count is bounded above byμ t . Loweringtcan reduce the exclusive count by introducing competitors, whereas maximum-cardinality counts cannot decrease. C.3 P1: Persistence across SAE widths We apply the independent pairwise matching and source-set intersection procedure in Section C.2.3 separately within Gemini and Nemotron, using each source SAE below width 131,072 and every larger trained SAE as targets. We report counts and source-width-normalized proportions. The largest SAE is a target only, since it has no larger comparison dictionary. Figure 2a uses the signed decoder-cosine metric in Section C.2.1. C.4 P2: Shared features across training distributions Figure 2b compares dictionaries trained onfull,wiki,no_wiki,rand1, andrand2, separately for each embedding model and dictionary family. SAE dictionaries contain 16,384 directions; PCA uses all d principal components, with d = 3072 for Gemini and d = 4096 for Nemotron. PCA dictionaries. We fit PCA separately for each embedding model and training mixture, using every embedding in that mixture. We stream the data to accumulate the countN, sums= P N i=1 x i , and second-moment matrix Q = P N i=1 x i x ⊤ i in float64, then compute ̄x = s/N,Σ = Q− N ̄x ̄x ⊤ N − 1 . We diagonalize this centered covariance matrix and order its orthonormal eigenvectors by decreasing eigenvalue. The PCA dictionary consists of these directions in the original embedding space; P2 uses the full d-component basis. PCA fitting uses the stored embeddings without additional length normalization or coordinate standardization. Dictionary matching. For each pair of training distributions, Figure 2b applies the metric in Section C.2. Exclusive direction matching and distribution-specific activation comparisons are reported in Section E. 40 C.5 P3: Hierarchical datasets and feature-recovery protocol For Figure 2c, we evaluate individual SAE features against binary category labels in three domains: word prefixes, geography, and organism taxonomy. The labels define hierarchical relationships, such as a two-letter prefix implying its first letter or a city’s country determining its continent (the organism taxonomy has more levels). English prefixes (prefix). We use the English frequency list from thewordfreqpackage. We retain the top 200,000 words in frequency order. Each normalized word is embedded on its own. The first character defines a parent label, and the first two characters define a child label for words of length at least two. We retain categories with at least 100 positive word rows across the full evaluation collection, giving 26 one-letter and 216 two-letter categories per embedding model. Geography (cities). We use all 69,182 records in the experiment’s GeoNamescities5000 snapshot, with country labels from each record’s ISO country code and continent labels obtained fromcountryInfo.txt. We retain 87 country and six continent categories with at least 100 city rows each. For the reported results, we embed two question variants of every city name:what country is [city] infor country prediction andwhat continent is [city] infor continent prediction, following the use of elicitation in Bhalla et al.[45]. (We found that without this elicitation, SAEs at no widths reliably could detect the continent.) Organism taxonomy (gbif). We use the GBIF Backbone Taxonomy. From itsTaxon.tsv andVernacularName.tsvtables, we select accepted species having at least one nonempty English vernacular name. This yields 123,495 species. We embed one scientific-name row per species and 259,286 deduplicated English common-name strings, leading to 382,781 name rows in total. Binary labels follow the species’ kingdom, phylum, class, order, family, and genus. We retain categories with at least 100 distinct species. This gives 4 kingdoms, 12 phyla, 33 classes, 131 orders, 276 families, and 43 genera. Once labels are constructed, F1 is evaluated over name rows, so species with more names can contribute more evaluation examples. Selection and test splits. We partition the data into selection and test sets. Word rows are divided equally within each two-letter-prefix stratum, with one-character words forming their own strata. City rows are divided equally within each country, and both question variants inherit this partition. GBIF partitions the underlying species into 61,748 selection species and 61,747 test species, with scientific names inheriting the species assignment. Common-name rows are assigned a single split, even when their species associations span both partitions. Every probe reads these saved split labels unchanged, and the same assignments are used across both embedding models and all widths. Exact reruns require these row tables and their source snapshots: the provided preparation scripts supply deterministic seed-1729 reconstructions for the word and city splits, but do not reproduce the historical row-by-row assignments. Fresh source downloads can also change the evaluation collections. Embedding and SAE evaluation. We compute gemini and nemotron embeddings for each input and compute activations for the nine full-mix SAEs for each model, with widths 512 through 131,072 and the corresponding TopK sparsities in Table 4. 41 Selecting a single-feature probe. For a binary target z ∗ i and SAE activations ˆz, define g j,τ (x) = 1ˆz j (x)≥ τ, τ > 0. For each coordinate active on selection rows, we sort its positive selection activations and evaluate every distinct observed value as an inclusive threshold. We choose the coordinate and threshold maximizing selection F1, requiring at least one selection true positive. For E =z ∗ i = 1 and F =g j,τ = 1, we use F1 D (E,F ) = 2 Pr D (E∩ F ) Pr D (E) + Pr D (F ) = 2TP 2TP + FP + FN on the relevant split. This evaluates how well a single thresholded feature identifies a category. Figure 2c details. Each point in Figure 2c is the unweighted mean of held-out test F1 over all retained categories at that level and both embedding models. The number of category–model pairs is twice the number of categories listed above, and the same pairs contribute at every width. The main-text a-prefix example, empirical Food and Music hierarchies, and fixed-sparsity recovery analysis are described in Section F. 42 D Extended Experiments for P1: Persistence across Size We extend the main persistent feature experiment in Figure 2a with controls for sparsity and training seed, activation-based matching, alternative matching criteria and similarity thresholds, and analyses of feature prevalence. D.1 Fixed sparsity At widths 512, 4,096, and 32,768, we replace each original source SAE by itsk= 128 counterpart, retaining every original larger-width target and using signed cosine at threshold 0.7 with the independent pairwise matching and intersection criterion in Section C.2.3. Table 6: Fixed-sparsity source replacement, using the same full-sweep target sets as the seed control. Only the source changes from its originalktok= 128; all original larger-width targets are retained. Same-width counts match the original and replacement source dictionaries. All comparisons use signed decoder cosine similarity ≥ 0.7; percentages divide by source width. Model Original width/k Same-width cross-sparsity matches Original source persistence k = 128 source persistence Gemini512/32389 (76.0%)303 (59.2%)221 (43.2%) 4,096/322,949 (72.0%)2,974 (72.6%) 2,405 (58.7%) 32,768/6416,466 (50.3%)15,881 (48.5%) 12,893 (39.3%) Nemotron 512/32410 (80.1%)259 (50.6%)210 (41.0%) 4,096/323,286 (80.2%)2,841 (69.4%) 2,607 (63.6%) 32,768/6418,198 (55.5%)12,750 (38.9%) 13,609 (41.5%) Table 6 shows persistence remains substantial, but is somewhat lower in general, especially for smaller widths. This seems to suggest that the conventional wisdom to increasekalongsidemis reasonable (and also is somewhat in alignment with our simulations Section G). D.2 Variation by seed We repeat the direction comparisons for gemini SAEs 512/32, 4,096/32, and 32,768/64 (width/k) with different training seeds. Table 7 shows a substantial number of matches across seeds, and that persistent remains similar (suggesting that persistence doesn’t vary heavily between SAEs). The same-width column uses pairwise maximum matching (Section C.2.3); the persistence columns use independent pairwise matching and intersection (Section C.2.3). Consistent with [20], there are a substantial number of different features learned when varying the random seed (especially for larger widths); simultaneously, there are many features that have matches even in all larger dictionaries. Table 7: Gemini direction stability across training seeds. Persistence uses the independent pairwise matching and intersection criterion. Width/k Same-width cross-seed matches Original source persistence Alternate source persistence 512/32470 (91.8%)303 (59.2%)293 (57.2%) 4,096/323,564 (87.0%)2,974 (72.6%)2,979 (72.7%) 32,768/6417,856 (54.5%)15,881 (48.5%) 15,998 (48.8%) 43 D.3 Exclusive matching Figure 6 repeats the source–target comparisons in Section C.3 using the exclusive persistence criterion in Section C.2.5. 5121K2K4K8K16K32K64K SAE width 0 5,000 10,000 15,000 20,000 Persistent feature count Gemini Nemotron 5121K2K4K8K16K32K64K SAE width 0% 20% 40% 60% 80% 100% persistent feature proportion Gemini Nemotron Figure 6: Persistent stability under strict exclusive matching at 0.7. A source direction counts only if it has a signed-cosine match above 0.7 in every larger trained dictionary, with no competing partner above 0.7 at either endpoint in each comparison. D.4 Persistence by activation similarity Figure 7 repeats the source–target comparisons in Section C.3 using signed Pearson correlation at threshold 0.7, with the metric and candidate-graph procedure defined in Sections C.2.2 and C.2.4. 5121K2K4K8K16K32K64K Source SAE width 0 5,000 10,000 15,000 20,000 25,000 30,000 Persistent feature count Gemini Nemotron 5121K2K4K8K16K32K64K Source SAE width 0% 20% 40% 60% 80% 100% Persistent feature proportion Activation Pearson (retained candidates) 0.7: independent pairwise persistence Figure 7: Persistent stability across SAE widths using activation similarity at signed Pearson correlation at least 0.7. Left: the number of source features matched in every independently computed pairwise maximum matching to a larger SAE on the retained candidate graphs. Right: this count divided by source width. Both panels compare Gemini and Nemotron, using the fixed tie-breaking procedure in Section C.2.3. 44 D.5 Sensitivity to the matching threshold We vary the matching threshold overt∈0.5,0.6,0.7,0.8using the independent pairwise matching and intersection criterion in Section C.2.3. Figure 8 uses signed decoder cosine similarity, and Figure 9 uses signed activation Pearson correlation. 5121K2K4K8K16K32K64K SAE width 0 5,000 10,000 15,000 20,000 25,000 30,000 Persistent feature count 5121K2K4K8K16K32K64K SAE width 0% 20% 40% 60% 80% 100% Persistent feature proportion Persistence across SAE widths · Signed decoder cosine t = 0.5t = 0.6t = 0.7t = 0.8GeminiNemotron Figure 8: Threshold sensitivity of direction persistence. 5121K2K4K8K16K32K64K SAE width 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 Persistent feature count 5121K2K4K8K16K32K64K SAE width 0% 20% 40% 60% 80% 100% Persistent feature proportion Persistence across SAE widths · Activation Pearson t = 0.5t = 0.6t = 0.7t = 0.8GeminiNemotron Figure 9: Threshold sensitivity of activation persistence. 45 D.6 Feature activation rates Figure 10 compares prevalence across SAE widths and KMeans cluster counts. For each SAE feature, we compute the fraction of the model’s full evaluation corpus on which its continuous post-TopK activation exceeds the SAE’s pooled median positive activation. This is one threshold per SAE, computed over all positive activation events across features and examples. For KMeans, prevalence is the fraction of examples assigned to each nearest centroid. We use Euclidean KMeans in the full embedding space, trained on every row of the corresponding full corpus for 100 total iterations, at 512, 4,096, 16,384, and 131,072 clusters. Both methods are evaluated on the same full corpus within each embedding family: 89,827,558 Gemini examples and 89,227,558 Nemotron examples. We plot kernel density estimates oflog 10 activation rates, using a bandwidth of 0.12 and scaling each curve to integrate to the number of features with positive rates. Zero-rate features are omitted from the logarithmic axis. The vertical axis uses a square-root scale with tick labels in the original density units. The total mass and average prevalence are partly constrained by dictionary width and sparsity. ForNexamples and an SAE of widthmwith TopK parameterk, there are at mostNkpositive activation events. Strictly exceeding their pooled median selects at most half of these events, with ties potentially reducing this number. Thus, ifp j denotes featurej’s above-median prevalence, then 1 m P m j=1 p j ≤ k/ (2m). For KMeans, cluster prevalences sum to one and have mean 1/m. Consequently, a growing low-prevalence tail and changes in the count-scaled density area partly reflect these constraints. The observed curve shapes provide descriptive evidence consistent with the recovery principle, rather than identifying atoms on their own. We observe a “nested” behavior, in which larger SAEs tend to include a similar number of coarse features as smaller SAEs, while also learning a longer tail of more granular features, consistent with the recovery principle. The KMeans prevalence distributions instead shift toward smaller clusters as the number of clusters increases. 46 0 22,500 90,000 202,500 Features per log₁₀ prevalence unit SAE (gemini)KMeans (gemini) −8−6−4−20 log₁₀ prevalence 0 22,500 90,000 202,500 Features per log₁₀ prevalence unit SAE (nemotron) −8−6−4−20 log₁₀ prevalence KMeans (nemotron) Number of features / clusters 512 1,024 2,048 4,096 8,192 16,384 32,768 65,536 131,072 Figure 10: Prevalence across SAE widths (left) and KMeans cluster counts (right), for Gemini (top) and Nemotron (bottom). SAE prevalence uses activation strictly greater than each SAE’s pooled median positive activation; KMeans prevalence uses nearest-centroid assignment. KMeans models are trained on the full corpus for 100 iterations. Colors indicate dictionary size. Curves show feature-count-scaled densities oflog 10 prevalence, with shared axes and a square-root vertical scale. Zero-rate features are omitted from the logarithmic axis. 47 To control for varying sparsities, Figure 11 repeats the comparison with TopK sparsity fixed at k= 128 for all SAE widths. We use widths 512, 4,096, 32,768, 65,536, and 131,072, with the same prevalence definitions, full evaluation corpora, and KMeans baselines as above. 0 22,500 90,000 202,500 Features per log₁₀ prevalence unit SAE, fixed k= 128 (gemini) KMeans (gemini) −8−6−4−20 log₁₀ prevalence 0 22,500 90,000 202,500 Features per log₁₀ prevalence unit SAE, fixed k= 128 (nemotron) −8−6−4−20 log₁₀ prevalence KMeans (nemotron) Number of features / clusters 512 4,096 16,384 32,768 65,536 131,072 Figure 11: Prevalence for SAEs with fixed TopK sparsityk= 128 (left) and KMeans (right), for Gemini (top) and Nemotron (bottom). SAE widths are 512, 4,096, 32,768, 65,536, and 131,072; KMeans cluster counts are 512, 4,096, 16,384, and 131,072. SAE prevalence is the fraction of examples with activation strictly above each SAE’s pooled median positive activation; KMeans prevalence is nearest-centroid assignment frequency after 100 full-corpus training iterations. Colors indicate the number of features or clusters. Curves show feature-count-scaled densities oflog 10 prevalence with bandwidth 0.12, shared axes, and a square-root vertical scale. Zero-rate features are omitted from the logarithmic axis. 48 D.7 Relationship between persistence and prevalence We directly compare feature prevalence with direction persistence using the median activation threshold from Figure 10. For an SAE with continuous post-TopK activationsˆz i (x), letτ SAE be the pooled median of all strictly positive activation events, across features and examples in its embedding model’s full evaluation corpus. We define p i (D) = 1 |D| X x∈D 1ˆz i (x) > τ SAE .(46) There is one threshold per SAE, rather than a separate median for each feature; the denominator includes every example. We use the full corpora for these measurements. For each source width in the original sweep, we comparep i for persistent features and non- persistent features. 0 10 −7 10 −5 10 −3 10 −1 Feature prevalence 0.0 0.2 0.4 0.6 0.8 1.0 Fraction selected in persistent matching Gemini 0 10 −7 10 −5 10 −3 10 −1 Feature prevalence Nemotron SAE width 512 1,024 2,048 4,096 8,192 16,384 32,768 65,536 Figure 12: Median-threshold feature prevalence and direction persistence under independent pairwise matching and intersection in Gemini (left) and Nemotron (right). Each curve represents a source width in the original sparsity schedule. Points group source features into up to ten prevalence-quantile bins, keeping ties together, and show the median prevalence and fraction selected in the persistent matching within each bin. All source features are included. Bin sizes can differ; lines connect descriptive bin summaries. The horizontal axis is logarithmic above 10 −8 with a linear segment retaining zero-prevalence bins. We measure the AUC of feature prevalence as a predictor of persistence. At widths 16,384, 32,768, and 65,536, AUCs are 0.814, 0.801, and 0.822 for Gemini, and 0.818, 0.905, and 0.917 for Nemotron. Restricting to features with nonzero prevalence gives respective ranges of 0.787–0.821 and 0.792–0.905. The association is weaker at the smallest widths. Overall, these results are consistent with the recovery principle’s prediction that the most prevalent atomic features are recovered. 49 E Extended Experiments for P2: Stability across Distributions We extend the direction-based comparison in Figure 2b using the exclusive matching criterion, activation-based matching, and the relationship between cross-distribution stability and cross- distribution prevalence. E.1 Cross-distribution stability under exclusive matching Figure 13 applies the exclusive criterion from Section C.2.5 to SAE and PCA dictionaries trained on different distributions, att= 0.7. This tests whether cross-distribution agreement persists when each match must have an unambiguous partner, allowing either sign for PCA. main wikipedia no_wikipedia random1 random2 0.460.370.490.46 0.460.250.460.40 0.370.250.380.40 0.490.460.380.41 0.460.400.400.41 SAE 0.080.010.070.07 0.080.010.150.04 0.010.010.010.03 0.070.150.010.05 0.070.040.030.05 PCA main wikipedia no_wikipedia random1random2 main wikipedia no_wikipedia random1 random2 0.470.360.510.52 0.470.270.570.48 0.360.270.380.47 0.510.570.380.50 0.520.480.470.50 main wikipedia no_wikipedia random1random2 0.000.000.000.00 0.000.000.030.00 0.000.000.000.01 0.000.030.000.01 0.000.000.010.01 0.0 0.1 0.2 0.3 0.4 0.5 matching feature proportion Gemini Nemotron Figure 13: Cross-distribution SAE and PCA matches under the alternative exclusive criterion at t= 0.7, using signed cosine for SAEs and absolute cosine for PCA. Each counted pair exceeds the threshold and has no other above-threshold partner at either endpoint. Entries divide the pair count by the dictionary width. Compared with the maximum-cardinality matching in Figure 2b, this criterion discards all pairs involving competing above-threshold partners. It therefore gives a stricter measure of unambiguous agreement between dictionaries. 50 E.2 Cross-distribution stability by activation similarity Figure 14 applies the activation metric, pairwise matching rule, and normalization in Section C.2 to the width-16,384 SAEs trained onfull,wiki,no_wiki,rand1, andrand2. Within each embedding model, the cell at rowDand columnD ′ evaluates both SAEs on all examples inD. Reversing the row and column changes the evaluation distribution, so the heatmap need not be symmetric. We construct sketches and candidate graphs for each row distribution using Section C.2.4, with moments computed over that distribution. Matched proportions range from 22.0% to 48.9% for Gemini and from 29.3% to 60.7% for Nemotron. full wiki no_wiki rand1rand2 full wiki no_wiki rand1 rand2 Source SAE and evaluation distribution 44.3%33.8%47.9%45.3% 45.0%22.0%45.5%38.8% 37.9%22.6%38.9%40.1% 48.9%45.1%36.8%40.3% 46.5%37.9%38.6%40.4% Gemini SAE activations full wiki no_wiki rand1rand2 full wiki no_wiki rand1 rand2 49.9%39.8%55.9%55.7% 51.1%30.0%60.7%53.4% 41.5%29.3%45.7%51.3% 56.8%60.0%44.9%56.1% 56.6%52.4%50.5%56.3% Nemotron SAE activations 0% 10% 20% 30% 40% 50% 60% 70% Matched features (Pearson ≥ 0.7) Comparison SAE training distribution Figure 14: Cross-distribution SAE activation matching for Gemini (left) and Nemotron (right). Each cell reports the maximum number of one-to-one matches at signed Pearson correlation≥0.7, divided by the SAE width of 16,384. Both SAEs are evaluated on the row’s distribution; columns identify the comparison SAE’s training distribution. 51 E.3 Feature prevalence and cross-distribution matching We test whether features matched between distribution-specific SAEs are more prevalent in both distributions, using the median-threshold definition in Equation (46). Proposition 2 suggests that the atomic features that are prevalent in both data distributions are recovered. For each width-16,384 source SAE trained onD, we compute its pooled median positive activation on the embedding model’s full evaluation corpus, including MIRACL. We freeze that single threshold when evaluating the source SAE on bothDand the target training distributionD ′ , and summarize prevalence by b i (D,D ′ ) = minp i (D),p i (D ′ ).(47) Both frequencies refer to the same feature in the same source SAE, including when it has no match in the target SAE. This avoids assigning a partner to unmatched features. Reversing source and target evaluates a different SAE, so the prevalence analysis is directional. We use the disjointwiki/no_wikiandrand1/rand2dataset partitions from Section C. These evaluation pairs exclude MIRACL, although threshold calibration uses the full corpus. They contain respectively 42,166,282/20,089,125 and 37,336,279/24,919,128 examples; they partition datasets rather than equal numbers of rows. The outcome is membership in one optimal SAE direction matching under the protocol in Section C.4. 0 10 −7 10 −5 10 −3 Minimum prevalence in both distributions 0.0 0.2 0.4 0.6 0.8 1.0 Fraction selected in direction matching Gemini 0 10 −7 10 −5 10 −3 Minimum prevalence in both distributions Nemotron Source → target Wiki → Non-Wiki Non-Wiki → Wiki Random 1 → Random 2 Random 2 → Random 1 Figure 15: Median-threshold prevalence in both distributions and cross-distribution direction matching for Gemini (left) and Nemotron (right). Curves identify the source and target training distributions. The horizontal coordinate is the bin median of the minimum prevalence of a source feature across the two distributions; the vertical coordinate is the fraction selected in a one-to-one direction matching. Quantile binning preserves ties and includes all source features, with the same axis convention as Figure 12. Higher prevalence in both distributions is associated with matching in all eight directional comparisons. AUCs, defined as in Section D.7, range from 0.864 to 0.923 for Gemini and from 0.839 to 0.927 for Nemotron. They remain large after restricting to features with nonzero prevalence in both distributions (ranges 0.844–0.905 and 0.828–0.921, respectively). 52 512 4,096 32,76865,536 131,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Mean held-out test F1 Prefix 1-letter 2-letter 512 4,096 32,76865,536 131,072 SAE width Cities Continent Country 512 4,096 32,76865,536 131,072 SAE width GBIF Kingdom Phylum Class Order Family Genus Fixed k=128 Figure 16: Hierarchical feature recovery at fixedk= 128. Mean held-out test F1 for Prefix, Cities, and GBIF at five SAE widths, pooling Gemini and Nemotron. Each curve averages the same category–model pairs as the corresponding curve in Figure 2c; each category’s feature and threshold are selected using training rows only. F Extended Experiments for P3: Hierarchical Feature Recovery We complement the aggregate hierarchy results in Figure 2c by controlling for sparsity, examining recovery of individual features and parent-child pairs, examining the prefix recovery examples, and discussing hierarchy extraction more broadly. F.1 Hierarchical recovery at fixed sparsity Figure 16 repeats the mean-F1 analysis of Figure 2c withk= 128 at widths 512, 4,096, 32,768, 65,536, and 131,072. We use the additional fixed-kcheckpoints at the first three widths and the existingk= 128 checkpoints at the final two. The datasets, eligible categories, train/test splits, detector selection, and averaging over category–model pairs are the same as in the main analysis. F.2 Counts and proportions of recovered categories and adjacent pairs To complement mean F1, Figures 17 and 18 count categories and parent–child pairs (in gbif, this means adjacent levels of the hierarchy) whose held-out test F1 is strictly greater than 0.6. F.3 Recovery of letter-prefix hierarchies The upper panel of Figure 3a shows Gemini test F1 for theaparent and its eligible two-letter children, selecting a separate detector for each. The lower recall heatmap instead fixes the selection-chosena detector at each width and measures its recall within each prefix group. That descriptive heatmap uses the full 200,000-word collection, including selection and test rows, rather than only the held-out split. Figure 19 extends the upper F1 panel to the 25 one-letter prefixes with eligible two-letter children, excludingx(since it does not have any qualifying two-letter children), using the same saved probes, nine widths, and category eligibility rules. These curves report continuous held-out test F1 without filtering by F1. We see more instances of mesoscale splitting (a U-shaped parent F1 curve). Figure 20 repeats this grid at fixedk= 128 for widths 512, 4K, 32K, 65K, and 131K, using the checkpoints described in Section F.1. 53 0 50 100 150 200 242 512 8,192 131,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Proportion recovered (F1 > 0.6) Word prefixes 0 20 40 60 80 93 512 8,192 131,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Geography 0 100 200 300 400 499 Number recovered 512 8,192 131,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Organism taxonomy GeminiNemotron Figure 17: Recovery of individual categories across SAE widths. A category is recovered when its selection-chosen detector achieves held-out test F1>0.6. Left axes show proportions; right axes show raw counts among 242 word prefixes, 93 geographic categories, and 499 organism categories. Geography uses different continent and country prompts. 0 50 100 150 216 512 8,192 131,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Proportion recovered (both F1 > 0.6) Word prefixes 0 20 40 60 80 87 512 8,192 131,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Geography 0 100 200 300 400 453 Number recovered 512 8,192 131,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Organism taxonomy GeminiNemotron Figure 18: Simultaneous recovery of adjacent parent–child pairs. Both selection-chosen detectors must achieve held-out test F1>0.6 and use distinct SAE coordinates within the pair. Left axes show proportions; right axes show raw counts among 216 prefix pairs, 87 continent–country pairs, and 453 adjacent-rank organism pairs. 54 0.0 0.5 1.0 a-prefixb-prefixc-prefixd-prefixe-prefix 0.0 0.5 1.0 f-prefixg-prefixh-prefixi-prefixj-prefix 0.0 0.5 1.0 k-prefixl-prefixm-prefixn-prefixo-prefix 0.0 0.5 1.0 p-prefixq-prefixr-prefixs-prefixt-prefix 512 8,192 131,072 0.0 0.5 1.0 u-prefix 512 8,192 131,072 v-prefix 512 8,192 131,072 w-prefix 512 8,192 131,072 y-prefix 512 8,192 131,072 z-prefix Gemini letter-prefix recovery SAE width Held-out test F1 Parent prefixMean child F1Individual children Figure 19: Gemini F1 recovery for 25 letter prefixes. Each panel is the analog of the upper panel of Figure 3a for one prefix fromatoz, excludingx, across the original nine SAE widths and sparsity schedule. Blue curves show parent F1; bold green curves show the unweighted mean F1 of eligible two-letter children; faint green curves show individual children. 55 0.0 0.5 1.0 a-prefixb-prefixc-prefixd-prefixe-prefix 0.0 0.5 1.0 f-prefixg-prefixh-prefixi-prefixj-prefix 0.0 0.5 1.0 k-prefixl-prefixm-prefixn-prefixo-prefix 0.0 0.5 1.0 p-prefixq-prefixr-prefixs-prefixt-prefix 512 4,096 32,76865,536 131,072 0.0 0.5 1.0 u-prefix 512 4,096 32,76865,536 131,072 v-prefix 512 4,096 32,76865,536 131,072 w-prefix 512 4,096 32,76865,536 131,072 y-prefix 512 4,096 32,76865,536 131,072 z-prefix Gemini letter-prefix recovery (k=128) SAE width Held-out test F1 Parent prefixMean child F1Individual children Figure 20: Gemini F1 recovery for 25 letter prefixes at fixedk= 128. The analog of Figure 19 at five SAE widths: 512, 4,096, 32,768, 65,536, and 131,072. Panels covera–z, excludingx. Blue curves show parent F1; bold green curves show the unweighted mean F1 of eligible two-letter children; faint green curves show individual children. Each category’s detector is selected using selection rows and evaluated on held-out test rows, with no F1 filtering. 56 F.4 Empirical extraction of feature hierarchies The food and music hierarchies in Figure 21 are curated subsets of a graph extracted from activation co-occurrences in the Gemini SAE with 131,072 features andk= 128. We use cached post-TopK activations onN= 89,827,558 corpus rows. A feature is active when its activation is strictly greater than 0.05; this threshold is the same for candidate parents and children. For a candidate parentP and childC, letn P andn C denote their activation counts andn PC their co-activation count. We estimate forward and reverse containment as bp P|C = n PC n C , bp C|P = n PC n P .(48) High forward containment indicates that the parent usually activates when the child does; low reverse containment indicates that the parent also covers examples outside the child. Candidate edges and directionality. We retain a candidate edgeP → Conly ifn C ≥100, n PC ≥ 100, n P > n C , and L P|C ≥ 0.70, bp C|P ≤ 0.50, bp P|C n P /N ≥ 5.(49) The last condition is a lift threshold: containment must be at least five times the parent’s marginal activation rate. The reverse-containment and support conditions orient edges from broader to narrower activation sets; the strict support ordering also rules out directed cycles. HereL P|C is the lower endpoint of the two-sided 95% Wilson interval for a binomial proportion. Writing bp = bp P|C , n = n C , and z = 1.96, it is L P|C = bp + z 2 /(2n)− z p bp(1− bp)/n + z 2 /(4n 2 ) 1 + z 2 /n .(50) This penalizes estimates based on fewer observations: for example, 90/100 and 900/1000 both give 90% containment, but their lower bounds are approximately 82.6% and 88.0%. We use the bound as a screening heuristic, since corpus rows can be duplicated or dependent and the interval does not adjust for testing many feature pairs. Removing ancestor shortcuts. After collecting all qualifying edges, we removeP → Cwhenever a featureMhas both qualifying edgesP → MandM → C. Thus, Food→Cheddar is removed if Food→Cheese and Cheese→Cheddar both qualify. Multiple parents can remain, and a broader ancestor edge can remain if an intermediate edge fails the thresholds. The procedure yields 154,316 qualifying ancestor pairs and 114,762 retained parent–child pairs. Selection and interpretation of the figures. Each hierarchy in Figure 21 displays 33 selected features and 32 retained edges. The compact main-text table in Figure 3b represents all 66 features and 64 parent–child relationships in three hierarchy-level columns, omitting only the feature IDs. We inspected cached activating texts to choose interpretable branches and assign descriptive labels; Tables 8 and 9 provide example texts. All edge statistics use the full corpus counts, rather than the sampled texts. The diagrams illustrate selected activation-containment relationships, not an exhaustive taxonomy or disjoint partition, and the labels are interpretations rather than independent ground truth. 57 Figure 21: Learned hierarchies for Music and Food. The labels here are automatically generated using Codex; activating examples for each feature in the hierarchy are given in Tables 8 and 9. 58 Table 8: Food hierarchy with five unique activating examples per feature. Food Foodf10077Food render • Food in England • Food Race • Food and Magic • Culinary diplomacy Cheesef59195チーズはいただき• Cheese (album) • Liste de mets à base de fromage • Premium Cheese! • Fromage à gogo Cheddar f21483Cheddar • CheddarU • Hombre de Cheddar • how cheddar cheese is made? • Cheddar (Käse) Mozzarella / related cheeses f29251 what is mozzarella • Mozzarella •모차렐라•موتزارلا•モッツァレッラ Blue cheeses f101286Bleue d’Auvergne • Roquefort • Bleu d’Auvergne • Gorgonzola • Fourme d’Ambert Gruyère / Emmental f20034 what is gruyere • Emmental de Savoie • what is gruyere cheese • 3. Jarlsberg. Jarlsberg is Norwegian variant of the cheese having holes and is a Swiss type cheese. This mellow and... • Emmental cheese Brie / Camembert f125850 Camembert • Coulommiers (Käse) • what does camembert cheese smell like • what’s in brie cheese • what gives brie unique taste Potatoesf36255Potato Project • Potatoes & You • when is potato day •Похороны картофеля• Jus de pomme de terre Potato cultivars f98628Lapin Puikula • Golden Wonder potato • Umatilla Russet • Lenape potato • Golden Wonder (potato) Mashed potatoes f94830how long will mashed potatoes stay good in the refrigerator? • do mashed potatoes go bad? • how long can mashed potatoes last? • how long are leftover mashed potatoes good for in the fridge? • do mashed potatoes go bad in the fridge? Baked potatoes f13508Check out this recipe for the Perfect Baked Potato: http://w.recipe4living.com/Recipes/Rec... It says to bake it for... • Jacket potato • how to check if baked potato done • Medium Plain Baked Potato: 150 Calories Size matters. A small baked potato (about 2 inches in diameter) has about 130... • how long is a baked potato good in the fridge Hash browns / rösti f26485 how to oven cook hash browns • Hash Browns • what is rosti potatoes •هاش براونز•뢰슈티 Tater tots f38138Tater Tots • With the lack of animal products not bearing any weight on making a decent batch of tots, tater tots are usually... • are tater tots bad for your diet? • are tater tots unhealthy for you? • are tater tots bad for your health? Fast foodf54683which fast food chain launched a ”value” campaign in 1988 that included drive-through • Wendy’s • gillian clare mcdonald is the ceo of which fast food chain • which fast food chain did paramount partner with to release movies at participating restaurants • Burger King Burger King f44349Burger King • Burger King advertising • Burger King Specialty Sandwiches • who started burger king • burger king hours for breakfast Wendy’s f49792The Wendy’s Company • Wendy’s • calories in wendys medium fries •وندیز• how many wendy’s are in florida Jack in the Box f80927Jack in the Box • what are in jack in the box tacos? • are jack in the box tacos made with meat? • what are jack in the box tacos made with? • are jack in the box tacos made with real meat? Hardee’s f119916Hardee’s • Hardee’s • The Carl’s Jr near me has this also. Since Carl’s Jr and Hardee’s are the same company they are probably exactly the... • hardee’s breakfast hours • are carl’s jr and hardee’s the same? McDonald’s burgers f13640 720 Cal. 720 Cal. Each Double Quarter Pounder with Cheese features two quarter pound* 100% fresh beef burger patties... • 510 Cal. 510 Cal. Each Quarter Pounder with Cheese burger features a ¼ lb. * of 100% fresh beef that’s hot,... • 510 Cal. 510 Cal. Each Quarter Pounder with Cheese burger features a ¼ lb. • 1250 Cal. 1250 Cal. Get double the fresh beef** flavor with a Double Quarter Pounder® with Cheese made with fresh beef... • Double Quarter Pounder®* with Cheese Meal. 1260 Cal. 1260 Cal. Get double the fresh beef** flavor with a Double... Rice dishesf112985Locrio •دمپختک• Galinhada • Pulao • Pelau Biryani f28414what vessel is biriyani cooked in • What is the main ingredient in biriani? • where is the movie biriyani being performed • Each community and region of India has its own traditional style of making biryani, some of the most famous styles of... • what is ulavacharu veg biryani? Nasi goreng f11153how long is the yangzhou fried rice • Nasi goreng pattaya • Yangzhou fried rice • 1 Mix and cook/fry the rice and egg together (in a wok is best). 2 Add the long green onion, chashaorou/bacon, carrots... • Yangzhou fried rice Yangzhou fried rice or Yeung Chow fried rice (Traditional Chinese:揚州炒飯; Simplified Chinese :... Fried rice f122816Arroz frito Yangzhou •揚州チャーハン•揚州炒飯• Nasi goreng pattaya • Nasi goreng tionghoa Sausagesf3953Port sausage •Список колбас• sausage • Chinese sausage • Garlic sausage Chorizo f22069what is chorizo used for • what is chorizo • Spanish chorizo and Mexican chorizo are very different from each other in looks, texture and taste. Since the Spanish... • Mexican Chorizo. 1 How it’s made and sold: Mexican chorizo is a spicy ground meat sausage that is most commonly sold... • Chorizo Salami f48955Salami Genoa • calories in hard salami lunch meat • how is salami processed • Yes, salami is very keto-friendly and low-carb. Salami has a very high-fat content and almost no carbs. Just be aware... • what is the difference between hard salami and dry salami? German sausages f110894 Wollwurst • Bratwurst • Regensburger Würstchen • what is bratwurst • Appenzeller Siedwurst Hot dogs f109777Cooking Instructions. Grill: Place over medium heat and turn until browned. Boil: Bring water to a boil, reduce heat,... • For just two or three hot dogs, put them in a shallow bowl with about 1/3 cup of water. Cover with plastic wrap. Be... • To cook hot dogs on the stove top, add up to 10 hot dogs to saucepan of boiling water. Return to boil; cover. Remove... • Ingredients: Mechanically Separated Turkey, Water, Salt, Contains 2% or Less Modified Food Starch, Potassium Lactate,... • How long to cook hot dogs. OSCAR MAYER Franks or Wieners can be cooked in just minutes on the grill. Heat the grill to... Nutritional information f50195 what is the nutritional value of flounder? • what is the nutritional value of eggplant? • Carbs: 45.6 grams. Fiber: 0.7 grams. Sodium: 399 mg. Folate: 24% of the Daily Value (DV) • The Basics. A 1-cup serving of fresh hearts of palm contains 4 grams of protein, 0.3 grams of total fat, 37 grams of... • what is the nutritional value of corn? Fast-food calorie questions f37858 how many calories in subway bacon egg and cheese flatbread? • how many calories in chicken noodle soup from panera? • how many calories in a burger king vanilla cone? • how many calories in a kfc donut chicken sandwich? • how many calories are in a mcdonald’s egg white? Calorie-breakdown template f38187 There are 111 calories in 1 cup of Cheerios. Calorie breakdown: 13% fat, 75% carbs, 12% protein. • There are 74 calories in 1 serving of Nescafe 3 in 1 Coffee. Calorie breakdown: 26% fat, 72% carbs, 2% protein. • There are 270 calories in a 1 link serving of Johnsonville Italian Sausage. Calorie breakdown: 73% fat, 4% carbs, 22%... • There are 80 calories in a 1/3 cup serving of Carnation Nonfat Dry Milk Powder. Calorie breakdown: 0% fat, 60% carbs,... • There are 320 calories in 1 serving of Taco Bell Nacho Fries. Calorie breakdown: 51% fat, 44% carbs, 5% protein. Percent Daily Value text f23538 * Percent Daily Values are based on a 2,000 calorie diet. Your daily values may be higher or lower depending on your... • Percent Daily Values are based on a 2000 calorie diet. Your daily values may be higher or lower depending on your... • Percent Daily Values are based on a 2,000 calorie diet. Your daily values may be higher or lower depending on your... • * Percent Daily Values are based on a 2,000 calorie diet. Your daily values may be higher or lower depending on your... • Percent Daily Values are based on a 2000 calorie diet. Your daily values may be higher or lower depending on your... Vitamins / minerals f120312 what vitamins and minerals are in red peppers? • which vitamins and minerals are in bananas? • what are the vitamins and minerals in cucumbers? • what vitamins and minerals does cabbage have? • what are the vitamins of mango? 59 Table 9: Music hierarchy with five unique activating examples per feature. Music Musicf33809مجله موسیقی•موسيقى إسبانيا• Diamond Music • Unduhan musik • BlueBeat Music Jazzf117052Kids: Live at Dizzy’s Club Coca-Cola • Sunset-Sunside • Sunset/Sunside • Robbins’ Nest: Live at the Jazz Showcase • Love and Peace: A Tribute to Horace Silver Charlie Parker / bebop f130651 Bird and Diz • Charlie Parker Memorial, Vol. 2 • Fats Navarro • Charlie Parker at Storyville • Groovin’ High (Dizzy Gillespie album) John Coltrane f86096Coltrane for Lovers •コルトレーン• Coltrane (Album) • Coltrane (1962 album) • John Coltrane Miles Davis f88107Miles Davis: The Complete Columbia Album Collection •ไมล์ส เดวิส•マイルス・デイヴィス•Дэвис,Майлз • Forever Miles Davis Mahavishnu / John McLaughlin f24353 マハヴィシュヌ・オーケストラ• Mahavishnu Orchestra • Mahavishnu Orchestra: The Complete Columbia Albums Collection • mahavishnu orchestra vision of the emerald beyond album • John McLaughlin Marsalis family f129158Branford Marsalis •Марсалис,Уинтон• Wynton Marsalis • Branford Marsalis Quartet •윈튼마살리스 Hip-hop / rapf25024Hip-hop indonésien • Liste von Hip-Hop-Musikern • Hip-hop espagnol • Something from Nothing: The Art of Rap • Hip hop Tuga N.W.A f94737NWA: Straight Outta Compton • N.W.A • N.W.A. • NWA • Straight Outta Compton: N.W.A 10th Anniversary Tribute Eminem f45908Eminem • what is the name of the movie that features eminem • Eminem Is Back • Discographie d’Eminem • My Son Marshall, My Son Eminem Snoop Dogg f1290Snoop Dogg has worked with Death Row Records. • Snoop Dogg released an album on Death Row Records. • Snoop Dogg • Snoop Dogg comes from California. • Snoop Dogg had a soundtrack released under Death Row Records. Tupac Shakur f123105Tupac Shakur • Tupac Shakur Legacy •توباك شاكور• what movie is about tupac shakur • who wrote the book about tupac shakur Lil Wayne f105791what is lil wayne’s record label called • Tha Carter I • what is lil wayne’s label called • what is lil wayne’s latest album called • lil wayne sorry 4 the wait album Country musicf22005Now That’s What I Call Country Volume 2 • This is a list of notable events in country music that took place in 2012. • 1988 in country music • My Life’s Been a Country Song • List of number-one country singles of 1982 (U.S.) George Jones f122748Discographie de George Jones • George Jones Sings His Greatest Hits •조지존스• how old was george jones when he died • George Jones Rascal Flatts f20266Discographie de Rascal Flatts • Rascal Flatts • who created rascal flatts music • what is the highest selling rascal flatts album • rascal flatts here comes goodbye album Patty Loveless / Vince Gill f421 Pam Tillis • Patty Loveless • Suzy Bogguss • Kathy Mattea • pam tillis all the good ones are gone album Hank Williams f22250hank wiiliams where did bocephus come from •هانك ويليامز• Hank Williams discography • what was hank williams first hit song •ハンク・ウィリアムズ Rock / metalf10590Phil Campbell and the Bastard Sons • Rhino Bucket • The Wildhearts • Taking Dawn • Ugly Kid Joe Hard-rock musicians / bands f61236 Dokken • Dave Meniketti • Jeff Pilson • Bobby Blotzer •ドン・ドッケン Nu / alternative metal f16925 Nonpoint • Spineshank • 40 Below Summer • Ill Niño • Stereomud Power / symphonic metal f123141 Ecliptica – Revisited: 15th Anniversary Edition • Sonata Arctica • Liste der Lieder von Sonata Arctica • The Metal Opera Part I • A Tribute to Sonata Arctica Doom / stoner metal f75431 Sleep’s Holy Mountain • Spirit Caravan • Electric Wizard • The Atomic Bitchwax • Scott Weinrich Electronic / dancef130808 Antoine Clamaran • Robbie Rivera • Sander Kleinenberg • Mauro Picotto • Ivan Gough Detroit techno cluster f5560 The Belleville Three • Juan Atkins • Kevin Saunderson • where did the belleville three come from • where did the belleville three first perform Trance mixes / compilations f111755 A State of Trance 2013 • A State of Trance • A State of Trance 2009 • A State of Trance 2012 • A State of Trance 2010 Basement Jaxx f17166Basement Jaxx • Discographie de Basement Jaxx • basement jaxx never say never album • when did basement jaxx album come out •ベースメント・ジャックス deadmau5 f35810what is the title of deadmau5’s fourth album • Deadmau5 Circa 1998-2002 • Full Circle (Deadmau5 album) • when does deadmau5 upload stuff i used to do • Mau5trap The Beatlesf18522The Beatles • Beatles Day •ビートルズ! •비틀즈• Beatles Stories Please Please Me / early Beatles f19564 The Beatles’ Please Please Me album was rush-released by Parlophone on 22nd March, 1963 to capitalise on the enormous... • Please Please Me (альбом) • Please Please Me • Please Please Me (album) • when was please please me released in uk Hey Jude f43221Hey Jude • where did the beatles film hey jude • what tv show featured the beatles singing hey jude • how many weeks did hey jude stay on the billboard hot 100 • Hey Jude (Wilson Pickett album) Let It Be / Get Back f94589 Let It Be • Let It Be (song) ”Let It Be” is a song by the English rock band the Beatles, released in March 1970 as a single, and... • Let It Be (песня) • Let It Be (song) • Let It Be (2004 film) Abbey Road / late Beatles f67785 Abbey Road (album) • Abbey Road (álbum) • The Best of Abbey Road • what is the title of abbey road’s last album • who sings lead vocals on best of abbey road 60 G Simulation experiments G.1 Recovery under width and sparsity variation We study recovery in finite synthetic distributions with known ground-truth atoms. These experiments assess whether trained TopK SAEs exhibit prefix recovery and how this behavior depends on the frequency distribution and the SAE sparsity. Figures 22 and 23 report approximate recovered prefix lengths, expressed in atoms and divided by SAE width. Data-generating distributions. Inputs lie inR 512 and are nonnegative linear combinations of exactlyK= 8 atoms. We draw each ground-truth direction independently from a standard Gaussian and normalize it to unit length. In the flat setting, there areM= 4096 atoms. An eight-element support S is sampled according to Pr(S) = Q i∈S i −α P T:|T|=8 Q j∈T j −α , |S| = 8, and its nonzero coefficients are independentUniform(0,1) draws. This is a product-weight distribution over fixed-size subsets; the power law specifies the support weights, rather than the marginal activation probabilities directly. In the hierarchical setting,M= 3072 atoms form 1024 families, each containing a parent and two children. We sample four families using the same product-weight law over family ranks, then activate each selected parent and one of its two children, chosen uniformly. The eight active coefficients are again independentUniform(0,1) draws. Thus a child can only occur with its parent, while either child can accompany the parent. Training and sweeps. We train nonnegative TopK SAEs at widthsm∈128,256,512,1024,2048,M. Generating sparsityK= 8 is fixed throughout;kdenotes the SAE’s activation budget. The frequency sweep usesα∈0.8,1.2,1.6,2.0,2.4atk= 8. The sparsity sweep usesk ∈1,2,4,6,8,10,12,16,32 atα= 1.6. Theorem 2 does not give guarantees forα= 0.8. We use three paired random seeds per setting, giving 468 distinct training trajectories after deduplicating the sharedα= 1.6,k= 8 setting. Each trajectory trains on 1,024M examples using Adam with learning rate 10 −3 ,β 1 = 0.9, β 2 = 0.99, and batch size 1024, with no learning-rate schedule. Decoder directions have unit norm and the decoder bias is fixed to zero. An auxiliary reconstruction loss with coefficient 0.25 targets up to 64 neurons inactive for 256 steps. Recovery criteria. Geometric recovery requires absolute cosine similarity at least 0.9 with a learned decoder direction. For hierarchical data, child comparisons project both directions onto the orthogonal complement of the true parent; recovery of a complete family requires three distinct decoder directions meeting the parent and two projected-child criteria. Activation recovery uses 200,000 independent validation examples to select coordinate thresholds and a maximum-cardinality injective assignment among atom–coordinate pairs with validation F1 at least 0.8. A matched atom is recovered if its F1 is also at least 0.8 on a separate set of 400,000 test examples. An activation-recovered family must recover all three members. For either criterion, letq i indicate recovery of atomiin the flat setting or complete familyiin the hierarchical setting, ordered by generating weight. We compute r = max 0∪ ( s : s X i=1 q i ≥ 0.9s )! . 61 128256512 1,0242,0484,096 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Prefix length / SAE width Flat · Geometry 128256512 1,0242,0484,096 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Prefix length / SAE width Flat · Activations 128256512 1,0242,0483,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Prefix length / SAE width Hierarchical · Geometry 128256512 1,0242,0483,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Prefix length / SAE width Hierarchical · Activations Simulated prefix recovery by distribution α= 0.8α= 1.2α= 1.6α= 2α= 2.4 Figure 22: Width-normalized prefix recovery across frequency distributions. Generating sparsityK= 8 and SAE sparsityk= 8 are fixed; each curve corresponds to a differentα. All models train on 1,024M examples. Rows show flat and hierarchical data; columns show geometric and activation recovery. The vertical axis is approximate prefix length in atoms divided by SAE width (three atoms per complete family for hierarchical data), with a 10% miss allowance. Lines show three-seed means and bands show seed ranges. 62 128256512 1,0242,0484,096 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Prefix length / SAE width Flat · Geometry 128256512 1,0242,0484,096 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Prefix length / SAE width Flat · Activations 128256512 1,0242,0483,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Prefix length / SAE width Hierarchical · Geometry 128256512 1,0242,0483,072 SAE width 0.0 0.2 0.4 0.6 0.8 1.0 Prefix length / SAE width Hierarchical · Activations Simulated prefix recovery by sparsity parameter k= 1k= 2k= 4k= 6k= 8k= 10k= 12k= 16k= 32 Figure 23: Width-normalized prefix recovery across SAE sparsities. Generating sparsity K= 8 and frequency exponentα= 1.6 are fixed; curves vary SAE sparsityk. All models train on 1,024M examples. Panels, normalization, and the 10% prefix miss allowance follow Figure 22. Lines show three-seed means and bands show seed ranges. 63 G.2 Persistent features across simulated SAE widths We apply the persistence metric in Section C.2.3. For each source widthm, we independently compute a maximum matching into each larger trained SAE and intersect the matched source-feature sets, holding the data distribution, SAE sparsity, and paired seed fixed. Matches require signed decoder cosine similarity or signed Pearson correlation of continuous post-TopK activations at least t= 0.8. The largest width (4096 for flat data, 3072 for hierarchical data) is a matching target only. We report the persistent-feature proportion, dividing the size of the intersection by allmsource features, including inactive features. We evaluate each group of six widths on the same 400,000 held-out test examples and compute all pairwise similarities without sketching or candidate truncation. Constant activation columns cannot form Pearson edges. The procedure in Section C.2.3 certifies each pairwise maximum on these empirical threshold graphs and verifies their matched source-set intersection. This measures persistence between learned SAEs. Figure 24 varies the frequency exponent; Figure 25 varies SAE sparsity. Atα= 1.6,k= 8, mean activation persistence across source widths ranges from 93.6–96.1% for flat data and 91.9–99.1% for hierarchical data. Hierarchical decoder persistence rises from 61.5% at width 128 to 91.8% at width 2048. Severe sparsity underspecification particularly reduces activation persistence, showing that persistent decoder directions need not imply persistent activation patterns. 64 1282565121K2K Source SAE width 0% 20% 40% 60% 80% 100% Persistent features (%) Flat · Decoder cosine 1282565121K2K Source SAE width 0% 20% 40% 60% 80% 100% Persistent features (%) Flat · Activation Pearson 1282565121K2K Source SAE width 0% 20% 40% 60% 80% 100% Persistent features (%) Hierarchical · Decoder cosine 1282565121K2K Source SAE width 0% 20% 40% 60% 80% 100% Persistent features (%) Hierarchical · Activation Pearson Simulated feature persistence by distribution α= 0.8α= 1.2α= 1.6α= 2α= 2.4 Figure 24: Persistent-feature proportions across frequency distributions att= 0.8. The intersection of source sets matched by independent maximum matchings to every larger trained width, divided by source SAE width. Generating sparsityK= 8 and SAE sparsityk= 8 are fixed; colors varyα. Rows show flat and hierarchical data; columns show signed decoder cosine and activation Pearson similarity. All models train on 1,024M examples. Lines show three-seed means and bands show seed ranges. The dashedα= 0.8 curve is the finite-distribution control outside the theorem’s guarantees. 65 1282565121K2K Source SAE width 0% 20% 40% 60% 80% 100% Persistent features (%) Flat · Decoder cosine 1282565121K2K Source SAE width 0% 20% 40% 60% 80% 100% Persistent features (%) Flat · Activation Pearson 1282565121K2K Source SAE width 0% 20% 40% 60% 80% 100% Persistent features (%) Hierarchical · Decoder cosine 1282565121K2K Source SAE width 0% 20% 40% 60% 80% 100% Persistent features (%) Hierarchical · Activation Pearson Simulated feature persistence by sparsity parameter k= 1k= 2k= 4k= 6k= 8k= 10k= 12k= 16k= 32 Figure 25: Persistent-feature proportions across SAE sparsities att= 0.8. The intersection of source sets matched by independent maximum matchings to every larger trained width, divided by source SAE width. Generating sparsityK= 8 and frequency exponentα= 1.6 are fixed; colors vary SAE sparsityk. Rows show flat and hierarchical data; columns show signed decoder cosine and activation Pearson similarity. All models train on 1,024M examples. Lines show three-seed means and bands show seed ranges. 66 dict. size iˆa i with molecule illustrationˆz i (cat)ˆz i (dog) ˆ Aˆz(cat) ˆ Aˆz(dog) 11 2 √ 6 a pet + 1 √ 6 a cat + 1 √ 6 a dog pet cat dog 3 √ 6 3 √ 6 2a pet + a cat + a dog 2 2 1 1 √ 2 a pet + 1 √ 2 a cat pet cat √ 20 a pet + a cat a pet + a dog 2 1 √ 2 a pet + 1 √ 2 a dog pet dog 0 √ 2 3 1 a pet pet 11 a pet + a cat a pet + a dog 2 a cat cat 10 3 a dog dog 01 Table 10: Learned directions illustrate molecular splitting. Corresponding learned features illustrate mesoscale feature splitting and absorption: at size 2 only, there is no learned feature that is active on both cat and dog. H A Molecular Model of SAE Scaling H.1 Computing the empirical molecules The empirical molecules in Tables 1 and 2 approximate smaller-SAE decoder directions using decoder directions from the Gemini SAE of width 131,072 and sparsityk= 128 as proxy atoms. These are decompositions of learned directions in embedding space; the activation values and recalls are measured separately by evaluating the corresponding SAEs. The 131K features serve as empirical proxies, rather than certified atomic directions. Below,ddenotes a target decoder row andb j a proxy-atom decoder row, both at their stored checkpoint scale. Prefix molecules: basis and least-squares fit. For Table 2, we use the word-prefix data, saved selection/test split, and single-feature probe procedure in Section C.5. For each of the 23 available one- and two-letter prefixes beginning witha, we take the 131K feature and positive activation threshold maximizing selection F1, without filtering by test F1. Their decoder rows form a matrix B= [b 1 ,...,b 23 ]. There are 22 distinct directions, sinceacandakselect the same feature; the regression retains both columns. For each selected smaller-SAE decoder direction, we compute c ⋆ = arg min c∈R 23 ∥d− Bc∥ 2 2 , using the minimum-norm solution, computed in double precision withnumpy.linalg.lstsqand rcond=None. There is no intercept, regularization, or nonnegativity constraint, and no additional normalization of the decoder rows. The fit uses all 23 basis columns, even though the table displays only the five prefixesa,ah,ab,ap, andas. Their proxy-atom feature IDs are 38618, 121657, 100965, 120576, and 32170, respectively. At each displayed width (512/k= 32, 2,048/k= 32, and 131,072/k= 128), we select the best probe for each of those five categories and deduplicate feature IDs. If a feature is selected for multiple categories, we use the threshold from the first category in the ordera,ah,ab,ap,as. For a category q, its recall cell is #x∈ T q : ˆz j (x)≥ τ j |T q | , 67 whereT q is the set of held-out words beginning withq. The four displayed child categories contain 70, 339, 292, and 456 test words, respectively. We display only rows whose recall is at least 0.5 on at least one category for which that feature was selected, using the same threshold across all cells in a row. This is a display filter using held-out recall; the probes and thresholds themselves are selected on the selection split. The displayed sums and circles retain only coefficients at least 0.015 among the five illustrated prefixes. Negative coefficients, smaller positive coefficients, and all other prefix components are omitted without refitting or renormalizing the retained coefficients. Circle radii are proportional to the cube root of the coefficient. Thus, these pictures summarize selected positive contributions rather than the complete signed least-squares reconstruction. At width 131,072, the illustrated directions are themselves basis columns, so their single-atom decompositions hold by construction; their held-out activation patterns are measured independently. Polysemantic molecule: sparse decoder reconstruction. For feature 3290 in the Gemini width-4,096,k= 32 SAE (Table 1), the candidate dictionary contains all 131,072 proxy-atom decoder rows. We use orthogonal matching pursuit (OMP). Starting with residualr=d, each step selects the unselected atom maximizing|⟨r,b j /∥b j ∥ 2 ⟩|, refits unrestricted signed least-squares coefficients on the selected support, and updates the residual. Atom normalization is used for selection; reported coefficients are expressed against the stored decoder rows. We retain the first support along this greedy path satisfying d− X j∈S c j b j 2 2 ≤ 0.05. This is an absolute squared-error threshold on the stored target row, not a relative-error criterion or a guarantee of globally minimal support. The retained fit has 17 terms, squared error 0.049297, and cosine similarity 0.975040 between the target and reconstruction. For readability, the diagram shows the five largest positive coefficients of this 17-term fit (also the first five atoms selected by OMP): 0.6763 on feature 8574, 0.1955 on 114331, 0.1665 on 68236, 0.1615 on 72509, and 0.0593 on 97729. These coefficients come from the full fit, not a refit restricted to five atoms, and the reported reconstruction error applies to all 17 terms. Circle areas approximately encode these positive coefficients; the smallest circle (97729) is enlarged for legibility. The other twelve signed terms are omitted from the illustration. Activation examples. The numerical cells in Tables 1 and 11 are cached post-TopK SAE activations on the same corpus inputs, with absent features assigned zero. They are not inferred from the decoder-decomposition coefficients. The upper block of the main table is manually selected from activating texts to illustrate different senses of transformation, including examples containing and not containing the substringtransform. The appendix gives twelve examples per proxy atom, curated across activation bins and semantic subtypes and sorted by decreasing activation. The lower block of the main table takes one example per atom from this appendix selection, with activation at least 0.09 on that atom and below 0.05 on each of the other four. These curated examples illustrate semantic selectivity rather than estimating its corpus frequency. Main-table shading uses a shared linear activation scale capped at 0.3. Decoder-similarity heatmaps. For Figure 26, we select the best probes for all 23 available a-prefix categories separately at widths 32,768/k= 64 and 131,072/k= 128, again without a test-F1 filter. After deduplicating feature IDs, each panel contains 22 directions. Entries are signed decoder 68 a a ab ac/ak ad ae af agah aial am anap ar as at au av aw ay az a a ab ac/ak ad ae af ag ah ai al am an ap ar as at au av aw ay az SAE width 32,768 a a ab ac/ak ad ae af agah aial am anap ar as at au av aw ay az a a ab ac/ak ad ae af ag ah ai al am an ap ar as at au av aw ay az SAE width 131,072 −0.4 −0.2 0.0 0.2 0.4 Decoder cosine similarity Figure 26: Decoder direction similarities among a-prefix probes. Signed pairwise cosine similarities for Gemini SAEs of widths 32,768 (k= 64, left) and 131,072 (k= 128, right). At each width, we take the best selected feature for each of the 23 available one- and two-letter a-prefix categories, without an F1 filter, and deduplicate feature IDs. This yields 22 unique decoder directions: acandakshare a probe at both widths. Rows and columns use the same prefix order in both panels, with a common color scale from−0.4 to 0.4. Diagonal self-similarities equal 1 and are shown at the maximum red. The median cosine over the 231 unique off-diagonal pairs is 0.1438 at width 32,768 and 0.0084 at width 131,072. cosines⟨d i ,d j ⟩/(∥d i ∥ 2 ∥d j ∥ 2 ), computed directly from the checkpoint rows, rather than from the molecule fits. The reported medians use the 231 unordered off-diagonal pairs in each panel. Table 11 gives twelve activating examples for each of the five 131K features in Table 1. 69 Table 11: Activating examples for the five 131K atoms. Each column contains twelve examples, sorted by decreasing activation. These appear more monosemantic than the feature in Table 1. 8574114331682367250997729 Transform(0.281)Transformers: Cybertron (0.288) is gewijzigd in of naar?(0.320)Transformer(0.273)* To multiply Row 1 by 3, use the elementary matrix: “‘ [3 0 0 0] [0 1 0 0] [0 0 1 0] [0 0 0 1]...(0.129) meaning of transformed (0.273) who does optimus prime fight in transformers autobots (0.270) تغیر(0.313)Transformer types(0.265)The concept lies in the fact that every elementary row operation can be represented as a matrix m...(0.117) Transformation (Politikwissenschaft)(0.265) Transformers: Prime(0.250)Cambios de imagen(0.265)how does transformer work (0.265) 1. Reflect about the y-axis: The new endpoints become (1, 4) and (-2, 3). <img>https://userupload... (0.117) when was the transformation theory introduced in quantum mechanics(0.236) what is the name of the god in transformers(0.245) when was the roman liturgical celebration changed(0.247) Why do transformers exhibit voltage ratios different from their specified values when connected i...(0.239) Following the steps from the previous example, we first find the reflection matrix for the line y...(0.111) To transform sqrt(2) into 1+frac(1)(sqrt(2)+1), we can follow these steps: 1. Start with ...(0.232) Transformers: The Score (0.237) how do i change my azure subscription?(0.239) The buzzing noise from the transformer is caused by the rapid fluctuations of magnetic fields, ty...(0.238) To rotate a vector around a point (2, 3) by 30 degrees, follow these steps: 1. Translate the vect...(0.108) what are the steps in graph transformation called (0.197) Transformers (film) voices of Ryan, Cullen, and Dunn, as well as Welker as Megatron. The Target c...(0.197) when did the grammy award for best traditional pop vocal album change(0.231) What are the consequences of mistakenly short-circuiting both the primary and secondary sides of...(0.228) The Hadamard gate represents a 180-degree rotation around the axis connecting the states |0⟩ and... (0.104) The transformations in the generating function are based on two key concepts: the derivative of a...(0.187) a unit commanded by Beast Wars Megatron, who ordered them to attack following the deaths of Optim...(0.197) If you want to change the unit vector vec a to vec a’ with a distance d’ from vec b, n... (0.188) How should I wire a center-tapped transformer with input voltages of 0V and 230V and output volta...(0.199) To obtain the graph of y=-2f(x) from the graph of f(x), we can perform the following transfor...(0.102) what is oncogenic transformation(0.186) Transformers: Exodus Cybertron where each individual determining their own path in life highly ap... (0.194) when did the west coast hockey league change to the east coast (0.177) what type of transformer is needed to achieve the nominal impedance of valve outputs (0.186) To represent the combined rotation of a 3D point (x, y, z), you can multiply the rotation matri...(0.102) A Reflection is a transformation in which the figure is the mirror image of the other. 1 Reflecti... (0.142) Greg “ M Sipher ” Sepelak is an American artist and writer for various Transformers comic ventu... (0.149) The 2015 expenditure for Army research, development and acquisition changed from 32 billion proj... (0.141) Commonly, transformers are used to increase or decrease the voltages of alternating current in el... (0.182) To transform a vector from System (1) to System (2) using intrinsic rotations, follow these s... (0.101) Is it correct to use Tietze transformations to transform a group into the group langle a, b, c...(0.121) Sentinel Prime(0.148)[’Open your Google Account. You might need to sign in.’, ’Under ”Security,” select Signing in to...(0.140) Control transformers are used to provide single point wiring when the control voltage differs fro...(0.149) In the context of rigid body transformations preserving the centroid at the origin, the transform...(0.052) At transform plate boundaries plates grind past each other side by side. This type of boundary se...(0.095) Barricade is the name of four fictional characters in the various Transformers universes.(0.139) The integrals span one full period of the DTFT, which means that the x[n] samples are also the co...(0.097) Earthing transformers are used to create a neutral point in a three-phase system, which provides...(0.140) Find the transformations applied to the function f(x)=frac1x to obtain the function y=dfr...(0.040) What is the correct interpretation of a Wick rotation, specifically regarding the transformation... (0.036) 에이수스이패드트랜스포머 (0.084) Constant Change Constant Change is the sixth studio album by Filipino singer-songwriter Jose Mari... (0.086) Will a step-down transformer work for using my US electronic devices (LED lights, string lights,... (0.137) How can I effectively visualize Lorentz transformations in the context of Minkowski spacetime? Sp... (0.018) 70 I An Atomic View of Feature Manifolds A growing body of work has observed that features do not necessarily exclusively have one-dimensional representations. For example, considering two-dimensional PCA projections of days of the year, one often recovers circular structure. The observation of such “manifolds” is sometimes used to argue against the linear representation hypothesis (which the atomic hypothesis strengthens). Here, we point out that the atomic hypothesis is not inconsistent with the observation of higher- dimensional non-linear structure. However, the atomic hypothesis can be viewed as implying that it should be possible to reconstruct this non-linear structure using atomic features. In particular, if we believe that the information represented byfis captured entirely byz, we should see the same non-linear structure arise inz. Looking exclusively atzdisentangles the information stored in the features themselves from information potentially stored by the directions corresponding to each feature. We consider subsets of SAE featuresS ⊆[131072] and consider PCA projections ofz S (x)∈ R M forx∈ X ∗ . We demonstrate that this procedure reveals geometric structure from atomic features. Notice this procedure only uses the learned featureszand not the learned directionsA. Importantly, this disentangles geometric structure arising from the embedding space (i.e., if different atoms are represented along correlated directions inR d ) from geometric structure arising purely from correlation in feature activations in data. We study two datasets: dates and colors. For each dataset, Figure 27 compares PCA of the raw embeddings with PCA of the continuous post-TopK activations restricted to a selected subsetSof the 131K Gemini SAE’s coordinates. The feature subsets and color filtering were chosen through exploratory inspection. Dates. The inputs are the 366 strings “[Month] [Day]” in calendar order, including “February 29”, with no year in the strings. We identify twelve month-related features in the gemini 131K SAE. We apply the full 131K encoder and TopK operation before extracting these twelve coordinates, assigning zero when a coordinate is not retained by TopK. PCA mean-centers the 366×3072 raw embedding matrix and the 366×12 activation matrix separately. The heatmap displays the same uncentered, continuous activation matrix in calendar order. Colors. The original collection contains 865 color-name rows with hex-string embedding inputs. We deduplicate hex codes case-insensitively, retaining the first row for each of the 765 distinct colors. After scaling RGB channels to [0,1], we retain colors with HSV saturationS ≥0.60, HSV value V ≥0.50, and HLS lightnessL ≤0.70. This leaves 374 colors. We sort these by increasing hue, decreasing saturation, and then increasing value. The eight selected 131K features were chosen from ranks 6, 7, 8, 10, 11, 12, 15, and 16, respectively, in the color collection’s feature-prevalence ranking at activation threshold 0.05. These corresponded to features that seemed color-related (rather than activating on hex patterns). That threshold is used only for selection: the plotted values are raw post-TopK activations. We refit three-component PCA separately to the 374×3072 raw embeddings and the 374×8 activation matrix using the full SVD solver, mean-centering without standardization, whitening, or additional normalization. These are fits on the retained colors, rather than subsets of PCA coordinates fitted to all 865 rows. The color heatmap uses the same raw activations, with rows ordered by activation-weighted circular mean position along the displayed hue sequence. Text and artwork examples. Table 12 shows five texts and five WikiArt images for each of the eight color features: 40 texts and 40 images in total. Texts are ranked by raw post-TopK activation 71 over all 89,827,558 Gemini training-corpus rows, removing exact duplicate full texts separately for each feature. Images are ranked independently by the same activation definition over all 116,261 rows of the cached WikiArt corpus. We take the five highest positive image activations per feature, with ties broken by decreasing image-vector index; no hex-color, style, or enrichment filter is applied to these images. WikiArt images are outside the SAE training corpus. These selected examples illustrate transfer beyond hex codes to multilingual text and images; they are not a held-out population performance estimate. 72 Figure 27: Manifolds arising from small sets of 131K gemini SAE activations. Top left shows PCA of raw embeddings, top right shows PCA of the SAE feature representation restricted to a subset of coordinates, and the bottom shows the activation pattern on these selected features. Importantly, the SAE-induced manifolds arise purely from correlation in activations, not correlation in feature directions. 73 top activating textstop activating artwork f124857 Оранжевый — хит сезона (0.4119) Orange Record Label (0.3469) Orange Hat (0.3398) Orange Link (0.3359) Orange ribbon (0.3344) (0.2292) (0.2244)(0.2110) (0.2063) (0.2052) f96080 Yellow hat (0.3367) Yellow Arrow (0.3251) Yellow monitor (0.3236) Yellow dog (0.3185) Yellow Hair (0.3169) (0.1967) (0.1800) (0.1798) (0.1796) (0.1787) f78457 Green hat (0.3725) Green and Silver (0.3619) Green Rose (0.3583) Green S (0.3556) Green Light (0.3518) (0.1837) (0.1826) (0.1816) (0.1791)(0.1762) f1397 Turquoise is not a color choice you make for your car. It chooses you! I’ve been... (0.2814) Turquoise ribbon (0.2587) 「ターコイズブルー」(Ｃ90 Ｙ20) (0.2533) turquoise number (0.2505) Colors That Go With Turquoise. ... Different shades of turquoise don’t have to match... (0.2317) (0.1757) (0.1655) (0.1618) (0.1565) (0.1511) f13055 Blue Rainbow (0.3702) Blue hat (0.3697) Blue Girl (0.3693) Blue Snowman (0.3691) Blue Monkey (0.3658) (0.1800)(0.1731) (0.1716) (0.1708) (0.1637) f129556 Purple Numbers (0.3519) Purple crocodile (0.3409) i feel like rich purple and gold are a match made in heaven and this reinforces that... (0.3329) Purple Love (0.3328) Purple Radio (0.3317) (0.2225) (0.2112) (0.1996) (0.1966) (0.1950) f71717 Pink Subaru (0.3593) Pink contract (0.3525) Pink Pineapple (0.3458) Pink and Gray (0.3367) Pink Man (0.3366) (0.2032) (0.1902) (0.1841) (0.1840) (0.1839) f57539 Красное на красном (0.4016) Red Elephant (0.3942) Red Bridge (0.3836) Red Post (0.3779) Red jersey (0.3765) (0.1869) (0.1832) (0.1821)(0.1800) (0.1792) Table 12: Top-activating text and image examples for eight color-selective features in the Gemini 131K sparse autoencoder. Each row shows the five highest-activating distinct general-corpus texts and five highest-activating WikiArt images for one feature, ordered by decreasing activation within each corpus. Parentheses give raw post-TopK activations. Texts are truncated for display. 74 J An Atomic View of the Platonic Representation Hypothesis If we interpret the rigidity property at an even higher level, then we can arrive at an even stronger theory: that atomic features are not atomic only with respect to an individual model, but are actually atomic as descriptions of the world (or at least, of descriptions of human-generated text). We might describe these atomic features as also being “platonic” [48]. So while there is no reason for atomic directions to be model independent (one can have an equivalent representation that is a rotation), it seems possible thatzcould be model independent, in which case different representationsf,f ′ are just different projections of the feature representation: that is, f = Az and f ′ = A ′ z. Same-width cross-model matching. We compare activations between models for all nine SAE widths from 512 to 131,072. At each width and thresholdt∈0.5,0.6,0.7,0.8, we apply the pairwise matching and candidate-graph procedures in Sections C.2.3 and C.2.4 to Gemini and Nemotron features. Figure 28 shows that the number of matched pairs increases with SAE width, while the fraction of each dictionary that is matched generally decreases. 5121K2K4K8K16K32K64K128K SAE width 0 5,000 10,000 15,000 20,000 25,000 Matched feature count Pearson 0.5 Pearson 0.6 Pearson 0.7 Pearson 0.8 5121K2K4K8K16K32K64K128K SAE width 0% 20% 40% 60% 80% 100% Matched feature proportion Pearson 0.5 Pearson 0.6 Pearson 0.7 Pearson 0.8 GeminiNemotron activation matching at equal SAE widths Figure 28: Feature-activation matching between Gemini and Nemotron SAEs of equal width. Curves give maximum-cardinality one-to-one matching at signed Pearson thresholdst∈0.5,0.6,0.7,0.8on 89,227,558 shared examples. Left: matched-pair count. Right: count divided by the common SAE width, including dead and unmatched features in the denominator. Counts are exact on the retained candidate graphs and lower bounds on exhaustive matching. The 131K comparison is included because matching across models does not require a larger SAE. Example matches. Tables 13 to 24 show twelve illustrative correlated pairs of features: two main examples and ten additional diverse pairs, with Pearson similarity ranging from 0.3 to 0.9. They cover clarification of meaning, algebraic transformations, education, trigonometry, astronomy, trade, programming, mechanics, food, sign languages, music, and botany. The captions describe qualitative interpretations of the activating texts. Each table contains fifteen sampled texts from each model: eight with source-feature activation in (0.05,0.1] and seven above 0.1. Gemini feature 71850 is an exception: it has no cached activations above 0.1, so all fifteen of its examples come from (0.05,0.1]. 75 Table 13: Clarifying the meaning or scope of a term. Pearson r = 0.352. TextGemini 71850 Nemotron 119331 Secara khusus, Chang’an Avenue hanya terdiri dari Chang’an Avenue Barat dan Chang’an Avenue T...0.0730.067 The term ”bile duct” is often used as a shorthand for the ”common bile duct,” but they are no... 0.0640.109 In both Great Britain and Ireland, the Continent is widely and generally used to refer to the...0.0640.118 Wenn in diesem Text oder in der wissenschaftlichen Literatur ”Mevalonsäure” ohne weiteren Nam...0.0640.037 While we often use the word ”sun” to refer specifically to our star, the Sun, the term ”sun”... 0.0580.128 Typically, ”abstraction” is used in the arts as a synonym for abstract art in general.0.0570.119 Kangaroo Island most commonly refers to Kangaroo Island in South Australia.0.0570.017 Stansted usually refers to London Stansted Airport. 0.0560.000 The Entombment usually refers to the entombment, or burial, of Jesus.0.0560.026 what word is often used casually to refer to both a database and the dbms used0.0560.077 Generally yes. When people refer to the company as Coke or as the more formal ”Coca-Cola” the... 0.0520.135 By extension, Himalayas is also the name of the massive mountain system which includes the Hi...0.0520.078 North Africa The term ”North Africa” has no single accepted definition. It is sometimes defin...0.0510.079 Some professional pool players still use the term billiards to describe what’s more commonly... 0.0510.130 Saat ini, istilah ”gubernur jenderal” umumnya digunakan lebih ke arah bekas koloni Kerajaan B...0.0500.076 Although the term progeria applies strictly speaking to all diseases characterized by prematu...0.0250.141 In an older and more restricted sense (not used here), the term chytrids referred just to tho...0.0440.121 The term combat (French for fight) typically refers to armed conflict between military forces... 0.0230.119 These terms are rarely used by the police themselves and instead are used by criminals, priso...0.0000.109 In U.S. usage, a ”plant condensate” is the equivalent of products classified as ”pentanes+” a...0.0210.104 While Mochica has been used in place of Moche by people describing this culture, the word Moc... 0.0000.101 Definition Challenges Since the terms ”risk appetite” and ”risk tolerance” often are used int...0.0000.100 [This agency, the Brain Injury Network, had asked them to change their definition, which form...0.0000.085 Blindness is frequently used today to describe severe visual decline in one or both eyes with... 0.0000.071 Hindi is an Indo-Aryan language or languages descended from Sanskrit , defined with various d...0.0230.071 I think the word is used in a electronics in a different way than what the word originally me...0.0000.067 The MENA acronym is often used in academia, military planning, disaster relief, and business... 0.0000.059 As a broad generality, in the United States, the SI metric system of units is predominantly u...0.0210.058 My son is 3 and I still consider him a toddler, but I was talking to a friend of mine who has... 0.0000.058 In computer science, this type of matrix representation is commonly known as a bit array, bit... 0.0000.057 Table 14: Whether algebraic transformations preserve solutions. Pearson r = 0.362. TextGemini 25799 Nemotron 102037 When we cancel out a common factor from both sides of an equation, we are essentially dividin...0.1240.128 When proving trigonometric identities, it is crucial to avoid manipulating them as if they we... 0.1210.098 Manipulating each side of a trigonometric identity separately is advised because it ensures t...0.1160.064 Dividing by zero is undefined in mathematics. If we divide both sides of an equation by a var...0.1150.132 Dividing both sides of a trigonometric equation by another trigonometric expression, like c... 0.1110.074 Explain why it is important to consider the possibility of losing solutions when dividing bot...0.1040.100 Which axioms or theorems in group theory support the justification of multiplying or adding a...0.1040.022 It is not accurate to say that there are solutions to a differential equation that cannot be... 0.0910.098 When you have the equation AC = B and wish to isolate C by multiplying both sides by A^(-1),...0.0770.042 Explain why the method of factorizing a variable to solve a quadratic equation is incorrect,...0.0730.084 Unfortunately, removing the floor function from both sides of the equation is not valid becau... 0.0700.043 If ab>0, then which one of the following must be true? (A) a/b>0 (B) a-b>0 (C)...0.0590.000 In the original question, it is mentioned that taking the natural logarithm of both sides of...0.0590.042 When we derive a new equation from a system of simultaneous equations, we are creating an equ... 0.0560.119 Why does the third elementary row operation, adding a row to another one multiplied by a numb...0.0540.023 When is it incorrect to square both sides of an inequality to solve it?0.0820.149 By substituting the expression for y from the original equation into one of the derivatives a...0.0000.134 The zero-product property states that if the product of two factors is zero, then at least on... 0.0000.122 Move all terms to one side of the inequality to simplify the expression.0.0410.108 When integrating a differential equation using separation of variables, why is it incorrect t...0.0340.107 Apply the product rule of differentiation to the potential parent term. If the result matches... 0.0000.106 how to solve equation with completing the square?0.0000.101 [’Isolate the x2u200b term on one side of the equal sign.’, ’Take the square root of both si...0.0250.072 In the proof of the solution to a recurrence relation using the characteristic equation, how... 0.0000.072 how to solve quadratic equations with variables?0.0000.067 To solve equations involving absolute values, follow these steps: 1. **Case Analysis**: Recog... 0.0130.063 [’Use the Rational Zero Theorem to list all possible rational zeros of the function.’, ’Use s... 0.0000.061 Suppose I have an equation f(x) = 0, and I know the solution exists. I perform a series of...0.0770.054 When calculating the Taylor expansion of a function that has different forms on either side o... 0.0000.051 As for the second restriction, commas are interpreted as or, so desmos is graphing values tha... 0.0000.051 76 Table 15: School activities and educational offerings. Pearson r = 0.516. TextGemini 31660 Nemotron 42453 Roanoke Rapids High School’s most prestigious club is the Student Council. Other clubs includ...0.1140.140 The Manhasset High School offers a variety of clubs and activities for our students. 0.1100.129 Riverdale Country School Abraham Award, which is an award given annually to high schools that...0.1100.108 Untuk menyalurkan bakat para siswa di berbagai serta melatih jiwa kepimpinan mereka, pihak se...0.1040.100 Popular extra curricular activities include National Cadet Corps (NCC), National Service Sche... 0.1030.129 Die Musical-AG führt regelmäßig Musicals auf, unter anderem: Winnfried (2006), Anything Goes...0.1020.049 সাধারণ পাঠঁপুƳেকর িশক্ষার বাইের সািহতঁ ও সংƯৄিত অijেন িবদঁালেয়র িশক্ষাথǶরা িবিভŭ...0.1000.076 softball, track and field (boys/girls), and cheer. Cave City students participate in competit...0.0890.100 what are some types of extracurricular activities?0.0830.064 productions, an annual talent show, and the annual ISAC Choir Festival. Schutz American Schoo... 0.0680.042 why is extracurricular activities important in school?0.0630.094 History of Georgetown University or in opposition to it in later years, such as the short liv...0.0600.021 and events. It has more than 400 student organizations and 17 intercollegiate sports teams. A... 0.0590.024 oldest still-running women’s college radio station in the country. In the 80s, it earned the...0.0570.052 The top 10 countries represented are China, Greece, Hong Kong, India, Turkey, Singapore, Cana...0.0510.021 Early Childhood Each class in the Early Childhood section has a designated teacher, fully qua...0.0300.149 A few examples of common educational opportunities that may be considered co-curricular inclu... 0.0990.145 Northland Preparatory Academy is a public charter high school and middle school in Flagstaff...0.0440.129 Wah Yan College, Hong Kong of Secondary Education Examination), Aestheic Education (Music and... 0.0530.125 in the school assembly are: The National Pledge is also recited. Special classes are provided... 0.0710.121 The Extended Student Services (ESS) Program extends Poway Unified School District’s education...0.0270.108 Today (4/19) your student will bring home a permission slip for a wonderful opportunity espec... 0.0300.105 Deer Valley High School offers all Arizona Interscholastic Association-sanctioned sports as o... 0.0680.092 Outside the classroom, Iona students can join more than 75 campus clubs, including about 10 f... 0.0450.092 Upon admittance to the school, students are given an academic level based on their proficienc...0.0240.091 Marianopolis is the only college that offers five special interest certificates: Environmenta... 0.0730.083 2.) give first priority to EOF activities over other extra-curricular activities. 3.) make ev...0.0270.059 Nord Anglia Education International Baccalaureate. In some locations, A-levels, the French Ba...0.0000.058 organization on campus with over 280 members. The band program incorporates a competitive cor... 0.0560.053 opportunity in which to engage youth and teachers in supporting English language-learning and...0.0190.053 Table 16: Trigonometric half-angle identities. Pearson r = 0.578. TextGemini 23469 Nemotron 114122 The half-angle identities can be derived from the sum and difference formulas for sine and co...0.1540.104 To find the exact value of sin(pi/6) using the half-angle identity, follow these steps: 1. St... 0.1310.064 How do you use the half-angle formulas to simplify the expression √((1+cos4x)/2)?0.1280.069 How do you use the half-angle identity to find the exact value of cos(pi/8) degrees?0.1250.077 How do you use a half-angle formula to find the exact value of tan((9π)/8)? 0.1160.072 Find the exact value of cos(105°) using the half-angle identity.0.1140.028 How do you use the half-angle identity to find the exact value of cosecant of 7pi/8?0.1130.076 How can we verify the identity #cos 2x = (1-tan^2x)/(1+tan^2x)# using the double angle identity? 0.0980.000 Given that cos A = frac(8)(17), find: a) sin 2A b) cos 2A Using the Pythagorean ide...0.0840.000 How do you find the value of sin(theta/2) given tan(theta) = 2 and 0 < theta < pi/2?0.0800.008 Starting with the double angle formula for tangent: tan(2x) = frac(2tan(x))(1-tan^2(x))... 0.0760.024 We can use the double angle identity for cosine to write: #cos(4x) = cos(2(2x))# #cos(4x) = 2...0.0730.013 The exact value of #sec(75°)# can be found as follows: 1. Note that #sec(75°) = frac(1)(cos...0.0660.029 To solve the equation #cos 2x = -frac(sqrt(3))(2)#, we can employ the double angle identity... 0.0650.000 Using the double angle formula for sine, we have: #sin(6theta) = 2sin(3theta)cos(3theta)# Usi...0.0590.019 The half-angle identities are trigonometric identities that express the sine, cosine, and tan...0.1420.114 To use the half-angle identities to find the exact values of sin 165°, cos 165°, and tan 165°...0.1370.113 The half-angle identities can be derived from the sum and difference formulas for sine and co... 0.1540.104 To derive the half-angle formulas for sine, cosine, and tangent, we can start with the identi...0.1550.104 Half-Angle Identities are trigonometric equations that relate the sine, cosine, and tangent o...0.1310.103 To find the exact value of trigonometric functions at specific angles using half-angle identi... 0.1340.103 To derive the half-angle formulas, we can use the following steps: * Use the double-angle for...0.1230.102 How do you use the half-angle formulas to find the exact value of tan(3π/8)?0.1360.077 How do you use the half angle identity to find the exact value of cos 101.25? 0.1210.074 The half-angle formula for sine is given by #sin^2(theta/2) = (1 - cos(theta))/2#. To simplif...0.0940.062 To simplify #csc ((7pi)/8)# using the half-angle formula, follow these steps: 1. Recognize th... 0.0980.062 To evaluate tan(-195) using the half-angle formula, we first need to find the value of tan(19... 0.0990.060 How do you use a half-angle formula to simplify tan[(9π) / 8]?0.1060.060 We can use the half-angle formula for cosine to find the value of cos(pi/5) without using a c... 0.0630.058 For the first question, we can use the half-angle formulas: sinleft(frac(x)(2)right) =... 0.0910.056 77 Table 17: Telescopes and astronomical optics. Pearson r = 0.604. TextGemini 64628 Nemotron 31241 what are the two key properties of a telescope and why is each important?0.2220.127 when was the vega bray telescope built 0.1700.040 Timeline of telescopes , observatories , and observing technology .0.1450.137 He therefore bought a telescope, which enabled him to watch as accurately as did the owner hi...0.1360.109 Can you see an American flag on the moon with a telescope? Even the powerful Hubble Space Tel... 0.1280.127 the version of the track recorded on ”Later... with Jools Holland” was included as a ”bonus l...0.1100.033 sensitivity is about 10 mJy. Steering of the antenna main lobe is accomplished with phase shi...0.1080.061 ملی ایران ساخت بنای ثابت محفظه تلسکوپ ۳٫۴ متری، محوطه سازی و ٔ طبق اعلام مجری رصدخانه ساختمان... 0.1000.069 Five hundred meter Aperture Spherical Telescope help change the focus to different areas of t...0.0990.097 The Square Kilometre Array ( SKA ) is a large multi radio telescope project aimed to be built...0.0940.033 from around 800 K (500 °C / 1000 °F) to 1500 K (1200 °C / 2200 °F) in just 6 hours. An observ...0.0790.041 Newsletter. Today’s BAFact: The Sun is 400,000 times brighter than the full Moon in the sky.... 0.0740.063 Moon missions. Russian Federation spacecraft is planned to send cosmonauts to the moon orbit...0.0700.016 Holders of a bachelor’s or a master’s degree in astronomy often enter an unrelated field. How...0.0640.044 are swfa scopes any good? 0.0570.057 As a rule you should avoid zoom binoculars, at least for astronomical use. Good zoom eyepiece...0.0230.203 I am planning to build a grism spectrometer based on F. Sigernes’ work. Can I use single lens...0.0000.173 On the other hand, a compound microscope has two sets of lenses, an ocular or eyepiece lens a... 0.0000.149 What online resources are available for amateur astronomy experiments, particularly those inv... 0.0800.123 Coronagraph A coronagraph is a telescopic attachment designed to block out the direct light f...0.0250.122 1 While looking at the objective lens and the stage from the side, turn the coarse focus knob...0.0000.116 از آینه های معروف جهان می توان به آینه ای که در تلسکوپ رصدخانه مانت پالومار کالیفرنیا کار گذ...0.1120.102 what is the name of the spherical lens placed on top of text to magnify the0.0000.077 [This root (tel-) means reaching the end (aim).. It is well-illustrated with the old pirate’s... 0.0910.073 Bernard Ferdinand Lyot ( 27 February 1897 , in Paris – 2 April 1952 , in Cairo ) was a Frenc... 0.0000.067 This was the observation that Venus has phases, much like our Moon does. To the naked eye, Ve...0.0710.060 lens system was made obsolete by later developments, primarily advanced by Panavision, Cinema... 0.0210.059 what year did the hubble space telescope not see neptune’s great red spot0.0330.058 what does the parabolic reflector of holophones radio produce0.0000.054 Lens Creek 0.0000.052 Table 18: Trade tariffs and protectionism. Pearson r = 0.651. TextGemini 61225 Nemotron 124167 Undang-Undang Smoot-Hawley0.1530.030 Loi Hawley-Smoot 0.1460.000 where were the first tariffs introduced in australia0.1450.088 what was the name of the tariff introduced by germany in 19250.1360.137 what did the republicans impose to protect themselves from foreign trade 0.1350.043 A commercial treaty is a formal agreement between states for the purpose of establishing mutu...0.1130.194 who was the president when steel tariffs were imposed0.1090.096 Customs duties in the United States 0.0860.097 Canadian-American Reciprocity Treaty primarily to the United States. The treaty was abrogated...0.0830.098 define autarky0.0800.000 Rückzoll 0.0720.000 the anti-cotton law league campaigned against the repeal of which laws0.0600.015 Aviation in Indonesia airlines by the end of 2015 or early 2016. Indonesia is considering ope...0.0570.039 what article did the jay treaty suspend in 1796 0.0540.013 United Nations Conference on Trade and Development The organisation grew from the view that e...0.0530.162 Special 301 Report and enforcement of intellectual property rights”. It also states that comp...0.0000.177 What are the main drawbacks of adopting a WTO Brexit (also known as a Clean Break)?0.0800.149 The so-called Really Good Friends (RGF) club is a sub-group of members of the World Trade Org... 0.0270.148 Interpreting NAFTA Rules of Origin. A rule of origin might contain. 1) a change in tariff cla...0.0430.145 of Canada in the 1920s. Its platform was entitled the ”New National Policy”, and it advocated...0.0890.129 Pollution haven hypothesis estimate the indirect effect of trade liberalization on pollution... 0.0360.129 Comprehensive and Progressive Agreement for Trans-Pacific Partnership favored that other coun...0.0230.118 Post-World War I economic expansion trade, GNP, and general quality of life. The high econom...0.0190.076 what act was passed in the kennedy round of trade negotiations 0.1070.074 __ INDEX __ Arun M. Kumar ( born 25 November 1952 ) has been the Chairman and Chief Executive...0.0000.072 Chile-United States Free Trade Agreement 0.0670.057 Products may come from many different countries – one part of a product may come from China... 0.0000.053 Both the United States and Japan belong to a variety of global organizations, including: the...0.0000.052 The International Jute Study Group ( IJSG ) is an intergovernmental organisation of states th... 0.0270.051 which international agreement provides for the protection of sound recordings 0.0000.050 78 Table 19: Comments in source code. Pearson r = 0.667. TextGemini 96133 Nemotron 76359 In Emacs, the provided examples actually use 2-character comment delimiters, which are extend...0.1670.178 Modeling line comments in a CFG can be achieved as follows: 1. If you’re not dealing with mul... 0.1580.171 how to comment out multiple lines in jupyter?0.1510.165 are comments passed to the parser?0.1500.082 Comment programming 0.1500.114 how to comment multiple lines in pycharm shortcut?0.1200.145 ulysses how to add comment?0.1020.022 How can I prevent syntax-propertize in Emacs from treating ’#’ within SPARQL URI’s as the sta... 0.0930.107 How can I make Yasnippet insert a comment sign on a new line when pressing Return? Q: When us...0.0890.108 জাভািưŷ এ েভিরেয়বল িডĤায়ার করা হয়varিক ওয়াডর্ িদেয়: var x; // defines the var...0.0740.044 Important As of Internet Explorer 10, conditional comments are no longer supported by standar...0.0680.029 Here document0.0580.023 A feature that fits into the former category is the extended properties of databases. Introdu... 0.0530.000 up vote 10 down vote favorite. 4. When I insert a comment or make a revision to some Microsof...0.0530.051 How to Query SQL Server 2000 for Stored Procedures Containing Text. For SQL Server 2000, the...0.0510.000 The recommended way to comment out multiple lines of code in Python is to use consecutive # s...0.1510.234 Multiline comment. A multiline comment starts with /* and can be found at any position in the... 0.1600.201 Multiline Comments in Bash Unlike most of the programming languages, Bash doesn’t support mul... 0.1460.172 how to comment out multiple lines in mysql workbench?0.1450.161 While there isn’t a direct way to insert line breaks in comments, they are designed to be con... 0.0700.145 There is also another way to document Python code using comments that start with ##. These ty... 0.1260.130 If the PyCharm IDE is used to write Python code - select multiple code rows to comment and pr...0.0900.113 Коментарий 0.0150.094 Looks like you are back with WordPress comments. I have sites with WordPress, DNN Blog, and D...0.0000.073 En kommentoi0.0210.069 VBdocman allows commenting and the automatic generation of technical documentation from Visua... 0.0770.065 There isn’t a built-in command in c-mode that mimics LaTeX’s latex-close-block for #ifdef blo...0.0530.064 This topic has been extensively discussed on various Meta sites within the Stack Exchange net...0.0510.053 Syntax highlighting for HTML code within a #+BEGIN_EXPORT HTML block in org-mode is affected... 0.0200.053 Commentaries dealing with the zahir (outward aspects) of the text are called tafsir, and herm...0.0000.051 Table 20: Mechanical bearings. Pearson r = 0.709. TextGemini 76787 Nemotron 40156 Bearing (mechanical) 1794. His was the first modern ball-bearing design, with the ball runnin...0.2030.282 Rolf Hoppler and Reinhold Errath evaluate the types of bearings used in the. cement industry,... 0.1910.184 A wheel bearing replacement can take anywhere from one to three hours depending on the type o...0.1570.195 Fluid bearing generators weighing hundreds of tons. They are also used in very heavy machiner...0.1460.218 Driving with 450+ answers. You didn’t indicate front or rear, or whether it is a front-wheel... 0.1370.153 SKF-89976A0.1080.097 The noise you’re experiencing could be due to a faulty main bearing in the front of the trans...0.1070.073 Consider the following diagram: [Image of a diagram showing Jack and Jill’s positions and the... 0.0980.031 Pacific Bearing Corporation guides, linear slides, linear motors and nuts for lead screws. In...0.0960.198 ACER Racing is a large OEM manufacturing company specializing in precision high performance r...0.0910.067 taken from the shafts of VFD-driven motors throughout the commercial bakery to illustrate the... 0.0750.085 1. Salvageability: The assembly can likely be salvaged with proper repairs. To verify the con...0.0730.128 Bearing an Hourglass is a fantasy novel by Piers Anthony. It is the second of eight books in...0.0640.092 A ship is anchored off a long straight shoreline that runs north and south. From two observat... 0.0570.000 Nicolas Fatio de Duillier ( also spelled Faccio or Facio ; 16 February 1664 – 12 May 1753 )...0.0520.000 Bearing (mechanical) rolling element bearing is a wooden ball bearing supporting a rotating t...0.2140.274 608 STANDARD The most common bearing size is the ”608.” Characterized by an 8m core, a 22m...0.1670.247 tapered roller bearing in combination with modern lubricants is extremely durable and is used... 0.1550.239 Slewing bearing. A slewing bearing or slew ring is a rotational rolling-element bearing or pl...0.1530.200 ABEC 5 bearings are the norm in skateboarding. You get a reasonable amount of speed, and at a...0.1770.174 Ball (bearing) the largest and smallest diameter measurement. For a given lot there is a ”lot... 0.0850.126 Oli pelumas harus selalu melumasi bagian-bagian poros engkol yang bergerak, hal ini tentunya...0.1040.117 when was the first roller bearing truck made0.2220.079 Tribology Online | Read articles with impact on ResearchGate, the professional network for sc... 0.0610.069 Cojinete magnético0.1990.060 • CHILDBEARING (noun). The noun CHILDBEARING has 1 sense: 1. the parturition process in human... 0.0430.059 Im August 1993 übernahm die Minebea alle Anteile der PM DM und wurde somit alleiniger Eigentü...0.0390.058 Footnote on Crankshaft Rebuilders. Did another search on Crankshaft Rebuilders. They use a ne... 0.0770.058 Tunnel crankcase there they were used in the more established diesel-electric designs. The fi... 0.0740.053 The behavior of an object rolling on a surface is influenced by the coefficient of rolling re...0.0290.052 79 Table 21: Butter. Pearson r = 0.788. TextGemini 59059 Nemotron 17701 which oil is used in sautéing to make butter0.2400.065 Grams in Butter Eight Tablespoon stick of unsalted butter weighs 110 grams. Therefore, 110 di... 0.1710.235 The rock gunnel or Butterfish(”Pholis gunnellus”) is an eel-like fish found in the intertidal...0.1630.132 Basically, cookies made with butter spread more and are flatter and crisper if baked long eno...0.1580.213 If a recipe calls for butter in ounces, simply divide the number of ounces by 4 to arrive at... 0.1540.244 is shea butter good for hair?0.1330.096 Hartkäse, Weichkäse und Butter sind relativ ergiebige Quellen für Vitamin K.0.1200.063 Fortunately for everyone who loves peanut butter, almond butter, and other nut butters, these... 0.0760.102 الجبنة القابلة لدهن )الكاسات(، شرائح الجبنة، الزبدة، السمن، كريمة الخفق، كريمة الطبخ، الموزا...0.0700.038 otto mønsted aarhus was the founder of which famous margarine0.0690.014 There are a number of these cleaners available on the Internet, most of them free to try out... 0.0640.025 Once melted, remove from heat and whisk in the hot sauce and salt. Coat the wings with the sa...0.0610.034 1 Bring a large pot of lightly salted water to a boil. 2 Add pasta and cook for 8 to 10 minut...0.0550.055 Directions. For the chocolate sauce: Combine the chocolate and 1/4 cup water in a heavy small... 0.0540.018 Prize (marketing) prizes from German margarine were molded in a light cream color designed to...0.0510.071 what kind of butter is fiddlehead fern pickle0.1690.208 when did the buttery become part of connecticut hall0.0360.144 Don’t use low fat spread-it might be better for you, but much of the fat is replaced by water... 0.0320.136 Place the butter cubes on top of the carrots. They will melt all over and do yummy things to...0.1230.132 West Buttercreek Airport0.0630.121 After 2 hours begin mopping the brisket with a butter/cajun seasoning mix that I call ”mop wa... 0.0770.107 Pan Frying. 1 1. Shuck the clams. Remove the clams from their shells if they are still intact...0.0810.104 Calories in Kroger Peanut Butter based on the calories, fat, protein, carbs and other nutriti...0.0340.086 do you have to refrigerate earth balance butter? 0.1520.085 Ingredients. 1 6-8 ounces salt pork, cut into 1/2 inch chunks. 2 4 Tbsp unsalted butter, divi...0.0700.080 A nervous stomach can give someone ”butterflies” in their stomach, or even make a person feel...0.0000.077 A Franzbrötchen (] ) is a small, sweet pastry, baked with butter and cinnamon. Sometimes othe... 0.0420.076 how many grams is 1 tablespoon of margarine?0.0660.064 You’re better off sticking with skim or low-fat dairy. Participants who ate whole-fat dairy (... 0.0460.060 Crème fraîche ( English pronunciation : -LSB- ˌkɹɛmˈfɹɛʃ -RSB- , -LSB- kʁɛm fʁɛʃ -RSB- , lit... 0.0420.055 Table 22: Sign languages. Pearson r = 0.802. TextGemini 61020 Nemotron 89987 hawaii sign language is an independent language from what other language0.1570.117 A free sign language worksheet for kids (a great reference when learning the ASL alphabet) AS... 0.1360.164 American Sign Language grammar signs: The hand shape is the initial of one of the English nam...0.1290.209 You could qualify by registering as a trainee sign language interpreter (TSLI). To register,...0.1180.185 Home sign is often the starting point for new deaf sign languages that emerge when deaf peopl... 0.1160.203 Lithuanian Sign Language ( LGK ) ( Lithuanian : Lietuvių gestų kalba ) is the national sign l...0.1090.187 who was the founder of indian sign language school in madras0.1030.056 Cuban Sign Language is the deaf sign language of Cuba . Little information is available . 0.0990.156 part. Thirteen of the most widely spoken indigenous languages are considered ”national langua...0.0730.132 who did jean massieu’s father play in sign gene0.0720.000 Fingerspelling Some writers have suggested that the body and hands were used to represent alp... 0.0710.112 Naga Hills Sign Language was a village sign language of India. Early in the 20th century, a h...0.0680.099 Project Nim (film)0.0580.000 Starbucks Malaysia’s Signing Store. Signing Store Workshop; Learn More . Policies; Looking fo... 0.0530.000 Deaf with the capitalized D represents people that are members of the Deaf Community, a lingu...0.0500.048 Black American Sign Language 2004 by Melanie Metzger and Susan Mather found that Black male s...0.1300.213 hard to draw. A language ’continuum’ is often described between signing with a strong sign-la...0.1170.205 ASLwrite () is a somacheirographic writing system that developed from si5s. It was created to... 0.0990.180 Many Australian Aboriginal cultures have or traditionally had a manually coded language, a si...0.0820.136 Rennellese Sign Language is an extinct form of home sign documented from Rennell Island in 19...0.0810.117 estimates for the number of people who use ASL are relatively easy to find in research and pr... 0.1080.109 can asl be used in other countries?0.1780.104 When the 26-year-old signed I’m proud of you, the president quickly signed back a Thank you....0.0810.099 Hanabiko ”Koko” (July 4, 1971 - June 19, 2018) was a female western lowland gorilla known for... 0.0810.086 She is now 13 months old. I started teaching her sign language as a tiny baby (signing as I t...0.0650.085 The World Trade Organization ( WTO ) is an intergovernmental organization that regulates inte... 0.0000.084 Tom Clerc Tom Clerc is a fictional deaf superhero appearing in Sign Gene, the world’s first f... 0.0600.071 As in other states, a certified interpreter in Idaho will command a higher fee for legal inte...0.0180.059 Centre for Deaf Studies, Bristol The Centre for Deaf Studies was a department of the Universi... 0.0460.054 The General Agreement on Trade in Services (GATS) is a treaty of the World Trade Organization... 0.0000.052 80 Table 23: Eurovision Song Contest entries. Pearson r = 0.865. TextGemini 123114 Nemotron 97094 Norway in the Eurovision Song Contest 19660.1750.124 United Kingdom in the Eurovision Song Contest 1968 0.1740.132 France in the Eurovision Song Contest 19560.1730.139 Denmark in the Eurovision Song Contest 19890.1690.144 Lithuania in the Eurovision Song Contest 2018 0.1470.142 Greece in the Eurovision Song Contest 20140.1380.157 Macedonia in the Junior Eurovision Song Contest 20130.1240.126 Singapore in the ABU TV Song Festival 0.0940.090 Bashkortostan in the Turkvision Song Contest 20130.0870.094 Venezuela in the OTI Festival0.0820.074 Kazakhstan in the Bala Turkvision Song Contest 2015 0.0780.093 France au Concours Eurovision de la chanson 19840.0720.026 Ukraine in the Eurovision Young Musicians0.0710.108 Eurovision Song Contest 1971 0.0560.051 Luxembourg au Concours Eurovision de la chanson 19610.0520.036 Switzerland in the Eurovision Song Contest 20090.1640.155 Denmark in the Eurovision Song Contest 20040.1640.144 Belarus in the Eurovision Song Contest 2005 0.1540.141 Netherlands in the Eurovision Song Contest 19830.1790.141 United Kingdom in the Eurovision Song Contest 19810.1770.135 Belgium in the Junior Eurovision Song Contest 0.1160.119 Ireland in the Eurovision Young Musicians0.1090.118 Wales in Eurovision Choir0.0410.088 Switzerland in the Eurovision Song Contest 2019 2018 contest would be selected through a Die... 0.0000.073 The Republic of Macedonia, participating under the name F.Y.R. Macedonia was represented at t...0.0000.065 Ukraine was represented in the Eurovision Song Contest 2005 by GreenJolly with the song ”Razo...0.0000.060 Montenegro Eurovision laulukilpailussa 0.0320.058 Spain in the Eurovision Song Contest 1965 Spain took part in the Eurovision Song Contest 1965...0.0000.054 Poland in the Early Middle Ages 0.0000.054 Eurovision Song Contest 1975 0.0600.052 Table 24: Astragalus and milkvetch species. Pearson r = 0.871. TextGemini 104756 Nemotron 51458 Astragalus campylanthus0.1660.165 Astragalus brauntonii 0.1650.151 Astragalus brachystachys0.1640.157 Astragalus annularis0.1630.160 Astragalus monanthemus 0.1600.155 Astragalus sprucei is a species of legume in the Fabaceae family. It is found only in Ecuador...0.1200.160 Astragalus sabulonum is a species of milkvetch known by the common name gravel milkvetch. It...0.1200.226 Astracantha gummifer 0.0910.075 Astragalus proimanthus is a species of flowering plant in the legume family known by the comm...0.0860.185 Astragalus cicer in regards to the inherent durability of the plant. In regards to obtaining...0.0810.143 The garage door gasket not only protects the bottom of the garage door from the floor each ti... 0.0750.000 purtuka or nakhuna (Astragalus Tribuloids),0.0720.039 Mesorhizobium sangaii is a bacterium from the genus Mesorhizobium which was isolated from roo...0.0670.056 Циклоастрогенол 0.0640.000 قتاد سمسمي0.0630.000 Astragalus bernardinus , known by the common name San Bernardino milkvetch , is a species of...0.1380.200 astragalus (aˑ·str ·gae·gä l) , s N latin: Names astragalus, Membranaceus astragalus; gummife... 0.1110.182 Astragalus cedreti ( also called Cedar astragalus ; ) is a species of flowering plant in the...0.1030.179 Astragalus crotalariae 0.1630.157 Astragalus newberryi0.1620.152 Astragalus pelecinus pelecinus0.1620.138 Es una planta herbácea perennifolia originaria de Norteamérica donde se distribuye por Estado... 0.1290.130 Astragalin is a chemical compound. It can be isolated from ”Phytolacca americana” (the Americ...0.1090.088 what is the herbage of astragalus molybdenus coated in 0.1400.086 Astragal 0.2070.083 why is the seed bank of astragalus nitidiflorus so sensitive0.1040.071 Chinese Herbs In Western View - Huang (Bei) qi (Radix astragali) Health Benefits and Side eff... 0.0800.067 where did the astragaloi perform divination in ancient greece 0.0820.060 locoweed toxins, pinpointing levels of toxicity in animals once they have ingested locoweed,... 0.0640.055 Astragalomancy pessomancy (also known as psephomancy) - another act of divination which uses...0.0530.054 81