Paper deep dive
Understanding sparse autoencoder scaling in the presence of feature manifolds
Eric J. Michaud, Liv Gorton, Tom McGrath
Models: Gemma-2-2b, Inception-v1
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/12/2026, 6:00:30 PM
Summary
This paper investigates the scaling behavior of Sparse Autoencoders (SAEs) when neural network activations contain multi-dimensional 'feature manifolds'. By adapting a capacity-allocation model from neural scaling literature, the authors demonstrate that when feature manifolds allow for gradual loss reduction (tiling), SAEs may exhibit pathological scaling where they allocate excessive latents to common manifolds at the expense of discovering rarer features. The authors provide theoretical conditions for this behavior and discuss its implications for interpretability.
Entities (4)
Relation Signals (3)
Sparse Autoencoder → exhibits → Scaling Laws
confidence 95% · The ability of SAEs to reconstruct activations follows scaling laws w.r.t. the number of latents.
Feature Manifold → influences → Scaling Laws
confidence 90% · feature manifolds (multi-dimensional features) influence scaling behavior.
Capacity-allocation model → models → Sparse Autoencoder
confidence 90% · we adapt a capacity-allocation model... to understand SAE scaling
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Sparse autoencoders (SAEs) model the activations of a neural network as linear combinations of sparsely occurring directions of variation (latents). The ability of SAEs to reconstruct activations follows scaling laws w.r.t. the number of latents. In this work, we adapt a capacity-allocation model from the neural scaling literature (Brill, 2024) to understand SAE scaling, and in particular, to understand how "feature manifolds" (multi-dimensional features) influence scaling behavior. Consistent with prior work, the model recovers distinct scaling regimes. Notably, in one regime, feature manifolds have the pathological effect of causing SAEs to learn far fewer features in data than there are latents in the SAE. We provide some preliminary discussion on whether or not SAEs are in this pathological regime in the wild.
Tags
Links
- Source: https://arxiv.org/abs/2509.02565
- Canonical: https://arxiv.org/abs/2509.02565
Trouble viewing inline? Open PDF directly →
Full Text
41,341 characters extracted from source content.
Expand or collapse full text
backref=true, list-name=End Notes Understanding sparse autoencoder scaling in the presence of feature manifolds Eric J. Michaud†,‡ Liv Gorton† Tom McGrath† †Goodfire ‡Massachusetts Institute of Technology Correspondence: ericjm@mit.edu Abstract Sparse autoencoders (SAEs) model the activations of a neural network as linear combinations of sparsely occurring directions of variation (latents). The ability of SAEs to reconstruct activations follows scaling laws w.r.t. the number of latents. In this work, we adapt a capacity-allocation model from the neural scaling literature (Brill, 2024) to understand SAE scaling, and in particular, to understand how feature manifolds (multi-dimensional features) influence scaling behavior. Consistent with prior work, the model recovers distinct scaling regimes. Notably, in one regime, feature manifolds have the pathological effect of causing SAEs to learn far fewer features in data than there are latents in the SAE. We provide some preliminary discussion on whether or not SAEs are in this pathological regime in the wild. 1 Introduction Sparse autoencoders Ng et al. (2011); Cunningham et al. (2023); Bricken et al. (2023); Gao et al. (2024); Rajamanoharan et al. (2024); Bussmann et al. (2024); Fel et al. (2025); Muchane et al. (2025); Costa et al. (2025) and related methods Dunefsky et al. (2024); Lindsey et al. (2024a); Gorton (2024); Minder et al. (2025); Ameisen et al. (2025) decompose neural network activations into a collection of sparsely activating latents. As SAEs have been scaled (now to millions of latents) Templeton et al. (2024); Gao et al. (2024); Lieberum et al. (2024); Lindsey et al. (2025), they have exhibited scaling laws Templeton et al. (2024); Lindsey et al. (2024b); Gao et al. (2024), where loss improves predictably as a power law with the number of latents in the SAE. Although sparse autoencoders learn many interpretable features, they may miss important structure in neural representations, either because there are an extraordinary number of very rare features in activations Olah and Jermyn (2024) or because the SAE architecture and training objective make incorrect assumptions about the structure of neural representations Csordás et al. (2024); Hindupur et al. (2025). In this work, we develop a formal analysis of SAE scaling behavior and scaling laws. We are particularly interested in understanding SAE scaling when activations contain a particular kind of structure: feature manifolds (multi-dimensional features) Engels et al. (2024a); Modell et al. (2025), and in whether feature manifolds could cause pathological scaling and exacerbate the problem of interpretability “dark matter” Olah and Jermyn (2024); Engels et al. (2024b). Our main approach is to adapt a mathematical model of neural scaling from Brill (2024) Brill (2024), where models allocate capacity between different data manifolds, to the case of SAEs. Guided by our model, we then conduct experiments to probe whether SAEs may scale pathologically in practice. 2 A model of sparse autoencoder scaling 2.1 The structure of data and the SAE architecture Activation data: We assume the multi-dimensional linear representation hypothesis Elhage et al. (2022); Engels et al. (2024b) and that neural network activation vectors ∈ℝdx ^d are generated as a sum of sparsely occurring features: =∑iix= _iS_if_i where iS_i is the subspace where feature i lives, specified by a d×did× d_i matrix whose columns are basis vectors of the subspace, and if_i is a random variable taking values in ℝdiR^d_i, where did_i is the dimension of feature i. Each feature if_i is sparse, supported on a small fraction of the data: pi=Pr[i≠]≪1p_i= [f_i 0] 1, so each activation vector is a sum of only a small subset of all features. SAEs: Sparse autoencoders attempt to reconstruct activation vectors as a sum of sparsely activating latents, consisting of an encoding step ^=Enc()=σ(e+e) f= Enc(x)=σ(W_ex+b_e) where σ is a nonlinearity and ^∈ℝN f ^N, and a decoding step ^=Dec(^)=d^+d x= Dec( f)=W_d f+b_d. Sparse autoencoders are trained with SGD on a loss ℒ=‖−^‖22+λS(^)L=\|x- x\|_2^2+λ\,S( f) where S(^)S( f) is a sparsity-encouraging loss like ‖^‖1\| f\|_1 or ‖^‖0\| f\|_0. 2.2 Assumption: SAE optimization reduces to a latent allocation problem Figure 1: SAE scaling on a toy feature manifold S1S^1. We train ReLU SAEs with an L1 sparsity penalty to reconstruct points on the circle S1⊆ℝ2S^1 ^2. We find that SAEs can slightly reduce their total loss by “tiling” the manifold with more sparsely activating latents. For SAEs with 4, 8, and 24 latents, we show the data (S1S^1) in grey, the SAE’s reconstruction in black, the decoder latent directions as arrows, and indicate the part of the circle that each latent fires on as colored arcs. If SAEs can reduce loss by “tiling” a manifold in this way, they may do this at the expense of learning rarer features. We assume that SAEs learn solutions where each latent j is specific to a particular feature i, so that ^j≠0 f_j≠ 0 only if i≠f_i 0. We further assume that the SAE latent decoder directions for the latents associated with feature i lie within the subspace iS_i, and that the subspaces where features live are orthogonal, i.e., i⟂kS_i _k if i≠ki≠ k. It follows that the SAE loss on any sample x is the sum of losses on samples where each active feature had fired alone: if ()A(x) are the “active” features i where i≠f_i 0 on x, then ℒ(x)=ℒ(∑i∈()ii)=assumption∑i∈()ℒ(ii).L(x)=L ( _i (x)S_if_i ) assumption= _i (x)L(S_if_i). We note that feature absorption and related phenomena Chanin et al. (2024) and the non-orthogonality of features in practice violate these assumptions, but we will accept them for the sake of expedience. If SAE latents are specific to features, then an optimal SAE’s loss on a feature i will be determined by the number of latents nin_i the SAE allocates to reconstructing feature i, which we denote Li(ni)L_i(n_i), and will be determined by the geometry of the feature if_i. The expected loss across activations x is then: [ℒ()]=[∑i∈()ℒ(ii)]=∑ipiℒ(ii)=∑ipiLi(ni)E_x [L(x) ]=E_x [ _i (x)L(S_if_i) ]= _ip_iL(S_if_i)= _ip_iL_i(n_i) We therefore reduce the SAE optimization problem to the problem of choosing how many latents nin_i to allocate to each feature i in the data. To understand SAE scaling behavior, we solve for the allocation of latents to each feature nin_i that minimizes ∑ipiLi(ni) _ip_iL_i(n_i) given a distribution over feature frequencies pip_i, the per-feature loss curves LiL_i, and the constraint on the total number of latents ∑ini=N _in_i=N. Figure 2: Left: Application of Brill’s (2024) Brill (2024) capacity-allocation model to SAE scaling. Right: numerical simulation of SAE scaling when the most frequently occurring feature is a manifold with loss scaling as L(n)∝n−βL(n) n^-β and all other features are discrete. We see that if β≪αβ α, then a simulated SAE with 100 million latents discovers only 3 million features. 2.3 Warm-up on discrete features To get comfortable with this formulation, we first apply it to the simplest case where all features are discrete. We imagine that for all features i, di=1d_i=1 and i=1f_i=1 on the fraction pip_i of samples where feature i is present and is 0 otherwise. For each feature i, an SAE incurs loss 11 if ni=0n_i=0 (reconstruction loss = 1 and sparsity loss = 0). If ni=1n_i=1, it can perfectly reconstruct the feature, incurring sparsity loss λ with L0 or L1 sparsity loss. Therefore L(n)=1ifni<1elseλL(n)=1\ if\ n_i<1\ else\ λ. With total loss ∑ipiL(ni) _ip_iL(n_i) and N latents in our SAE we see that the optimal solution is to allocate one latent to the most commonly occurring N features. In this setup, then, the effect of scaling SAEs is to learn an increasing number of discrete features in the data, in decreasing order of their frequency. We will often be interested in the number of features “discovered” by the SAE D(N)=|i:ni>0|D(N)=|\i\ :\ n_i>0\|. In this setting, D(N)=ND(N)=N. To recover the power law scaling that SAEs exhibit empirically, we only need to assume that pi∝i−(1+α)p_i i^-(1+α). With this assumption, the improvement in total loss from adding a marginal latent i follows pi(1−λ)p_i(1-λ), and integrating from i=1…Ni=1… N we get that the total loss drops off as a power law ℒ(N)∝N−αL(N) N^-α. We note that this model of SAE scaling mirrors the “quanta” model of neural scaling from Michaud et al. (2023), where here the “quanta” are features. This picture also agrees with the finding from Templeton et al. (2024) that SAEs learn features for concepts in data approximately in order of the frequencies (roughly Zipfian) at which those concepts occur. 2.4 Intuition behind pathological manifold scaling Recently, several works have commented on the existence of feature manifolds Modell et al. (2025); Gorton (2024) (multi-dimensional features) Engels et al. (2024a) in neural networks. In our formalism above, these are features i with di>1d_i>1 and where the range of if_i is a manifold embedded in ℝdiR^d_i. With feature manifolds, instead of having a discrete Li(ni)L_i(n_i) curve like above, LiL_i might drop off slowly as nin_i grows. To show that this is possible, in Figure 1, we show the scaling curve for ReLU SAEs (σ=ReLUσ= ReLU) trained with L1 penalty to reconstruct points on a circle ∈S1⊆ℝ2x∈ S^1 ^2. We observe that these SAEs can gradually reduce their total loss by more finely “tiling” the manifold with latents that activate more sparsely, and that this manifold can accommodate dozens of latents before loss plateaus. While such solutions ruthlessly minimize the SAE loss, it is not obvious that they would be better from an interpretability standpoint. Our fundamental concern is that if Li(ni)L_i(n_i) curves decrease gradually, it could be optimal for an SAE to tile common feature manifolds instead of discovering rarer features in data. Figure 3: ReLU SAE scaling on individual toy feature manifolds, showing how L(n)L(n) curves depend on feature geometry. We train with L1 sparsity λ=0.1λ=0.1. Left: SAE scaling on data drawn from a single unit hypersphere. Right: SAE scaling on data sampled in : 0.5<||<2\x\ :\ 0.5<|x|<2\. 2.5 Solution for power-law L(n) following Brill (2024) To model SAE scaling in the presence of feature manifolds, we assume that feature frequencies decay as a power law pi∝i−(1+α)p_i i^-(1+α). For mathematical simplicity, we also assume that all features have the same power law per-feature loss curve Li(ni)=ni−βL_i(n_i)=n_i^-β. In this setting, our model of SAE scaling directly corresponds to the model of neural scaling from Brill (2024) Brill (2024), where neural networks allocate units of capacity nin_i towards approximating functions on distinct power-law distributed data manifolds, with per-manifold loss scaling as ni−c/Dn_i^-c/D. We show similar derivations to Brill Brill (2024) in Appendix B and summarize the core results in Figure 2 (left). In our notation, there is a key threshold: α<βα<β. When α<βα<β, expected loss scales as ℒ(N)∝N−αL(N) N^-α like in the discrete feature case, and D(N)∝ND(N) N, so the “efficiency” at which SAEs discover features D(N)/ND(N)/N tends towards a constant. However, when β<αβ<α, then ℒ(N)∝N−βL(N) N^-β and D(N)∝N1+β1+αD(N) N 1+β1+α, and so D(N)/ND(N)/N tends towards 0. This illustrates the core dynamic of concern: when SAEs can continue to reduce loss by “tiling” common feature manifolds, then it can be optimal for them to do this at the expense of discovering other, rarer features. 2.6 Numerical simulation While our mathematical model above assumed that all features have the same Li(ni)L_i(n_i) curve, we can run numerical simulations with arbitrary heterogeneous Li(ni)L_i(n_i) curves. In Figure 2 (right), we conduct simulations where the first feature scales as n−βn^-β but all other features are discrete and scale as step functions n=01_n=0, and with pi∝i−(1+0.5)p_i i^-(1+0.5). We find that, with only a single feature with power-law L(ni)L(n_i) scaling, when β<αβ<α this feature begins to absorb the vast majority of latents in the SAE once N is large. 3 SAE scaling on synthetic features and on real neural networks In our analysis, whether SAEs will scale pathologically depends on the shape of their per-feature loss curves Li(ni)L_i(n_i). We showed one such L(n)L(n) curve in Figure 1, but we further explore this by training SAEs on a variety of synthetic manifolds in Figure 3 and in Appendix Figure 6. Overall, we find that the shape of the L(n)L(n) curve depends on the manifold geometry, with some manifolds accommodating many thousands of latents without saturating loss while on others the loss curve plateaus very quickly (effectively β→∞β→∞). In Section A.1, we provide further discussion on what α and β may be, and whether SAEs might be scaling pathologically, in practice. When a large number of SAE latents are allocated to a relatively low-dimensional feature manifold, we’d expect the cosine similarities between the decoder latents allocated to that manifold to be close to 11. In Appendix E, we show the distribution over decoder latent nearest neighbor cosine similarities for SAEs trained on LLM and vision model activations, and find some differences between them. 4 Discussion In this short paper, we have described a model of SAE scaling which reduces the SAE optimization problem to the problem of optimally allocating different numbers of SAE latents to different features in data. As in the model of scaling from Brill Brill (2024) SAE scaling laws either result from an underlying power law distribution over features pi∝i−(1+α)p_i i^-(1+α), or from the improvements in loss from tiling common feature manifolds following a power law Li(ni)∝ni−βL_i(n_i) n_i^-β. When β<αβ<α, SAE latents could massively accumulate on commonly occurring feature manifolds. Unfortunately, we do not resolve the question of whether SAEs are in this pathological scaling regime in practice. Our uncertainty is due to our knowing neither the distribution over true feature frequencies (determining α) nor the geometry of real-world neural network feature manifolds (which determines β). We give some additional commentary in Section A.1. Arguably our most important observation is that when we train SAEs on toy feature manifolds with variation in the radial direction, our SAEs do not tile the manifold and instead exhibit L(n)L(n) curves which plateau early. If this geometry is realistic Engels et al. (2024a); Olah (2024), then manifolds may not pose an issue to SAE scaling in practice. Overall though, we view this work as primarily about framing, rather than completely answering, this interesting problem. Acknowledgments and Disclosure of Funding We thank Owen Lewis, Michael Pearce, Jack Merullo, Dan Balsam, Michael Byun, Elana Simon, and Wes Gurnee for helpful conversations and feedback. EJM was supported by the NSF via the Graduate Research Fellowship Program (Grant No.2141064). This work was done at Goodfire AI in San Francisco. References Ng et al. [2011] Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1–19, 2011. Cunningham et al. [2023] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. Bricken et al. [2023] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html. Gao et al. [2024] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024. Rajamanoharan et al. [2024] Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.14435, 2024. Bussmann et al. [2024] Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410, 2024. Fel et al. [2025] Thomas Fel, Ekdeep Singh Lubana, Jacob S Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Wattenberg, Demba Ba, and Talia Konkle. Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models. arXiv preprint arXiv:2502.12892, 2025. Muchane et al. [2025] Mark Muchane, Sean Richardson, Kiho Park, and Victor Veitch. Incorporating hierarchical semantics in sparse autoencoder architectures. arXiv preprint arXiv:2506.01197, 2025. Costa et al. [2025] Valérie Costa, Thomas Fel, Ekdeep Singh Lubana, Bahareh Tolooshams, and Demba Ba. From flat to hierarchical: Extracting sparse representations with matching pursuit. arXiv preprint arXiv:2506.03093, 2025. Dunefsky et al. [2024] Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable llm feature circuits. Advances in Neural Information Processing Systems, 37:24375–24410, 2024. Lindsey et al. [2024a] Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layer features and model diffing, October 2024a. URL https://transformer-circuits.pub/2024/crosscoders/index.html. Research update. Gorton [2024] Liv Gorton. Group crosscoders for mechanistic analysis of symmetry. arXiv preprint arXiv:2410.24184, 2024. Minder et al. [2025] Julian Minder, Clément Dumas, Caden Juang, Bilal Chugtai, and Neel Nanda. Robustly identifying concepts introduced during chat fine-tuning using crosscoders. arXiv preprint arXiv:2504.02922, 2025. Ameisen et al. [2025] Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/methods.html. Templeton et al. [2024] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html. Lieberum et al. [2024] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147, 2024. Lindsey et al. [2025] Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of a large language model. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/biology.html. Lindsey et al. [2024b] Jack Lindsey, Tom Conerly, Adly Templeton, Jonathan Marcus, and Tom Henighan. Scaling laws for dictionary learning, April 2024b. URL https://transformer-circuits.pub/2024/april-update/index.html#scaling-laws. Circuits Updates — April 2024. Olah and Jermyn [2024] Chris Olah and Adam Jermyn. The dark matter of neural networks? Transformer Circuits Thread, July 2024. URL https://transformer-circuits.pub/2024/july-update/index.html#dark-matter. Part of Circuits Updates - July 2024, Anthropic Interpretability Team. Csordás et al. [2024] Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations. arXiv preprint arXiv:2408.10920, 2024. Hindupur et al. [2025] Sai Sumedh R Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba. Projecting assumptions: The duality between sparse autoencoders and concept geometry. arXiv preprint arXiv:2503.01822, 2025. Engels et al. [2024a] Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear. arXiv preprint arXiv:2405.14860, 2024a. Modell et al. [2025] Alexander Modell, Patrick Rubin-Delanchy, and Nick Whiteley. The origins of representation manifolds in large language models. arXiv preprint arXiv:2505.18235, 2025. Engels et al. [2024b] Joshua Engels, Logan Riggs, and Max Tegmark. Decomposing the dark matter of sparse autoencoders. arXiv preprint arXiv:2410.14670, 2024b. Brill [2024] Ari Brill. Neural scaling laws rooted in the data distribution. arXiv preprint arXiv:2412.07942, 2024. Elhage et al. [2022] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. https://transformer-circuits.pub/2022/toy_model/index.html. Chanin et al. [2024] David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Satvik Golechha, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409.14507, 2024. Michaud et al. [2023] Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36:28699–28722, 2023. Olah [2024] Chris Olah. What is a linear representation? what is a multidimensional feature? Transformer Circuits Thread, July 2024. URL https://transformer-circuits.pub/2024/july-update/index.html#linear-representations. Part of Circuits Updates - July 2024, Anthropic Interpretability Team. Till [2024] Demian Till. Do sparse autoencoders find “true features”? https://w.lesswrong.com/posts/QoR8noAB3Mp2KBA4B/do-sparse-autoencoders-find-true-features, 02 2024. Accessed: 2025-08-11. Anders et al. [2024] Evan Anders, Clement Neo, Jason Hoelscher-Obermaier, and Jessica N. Howard. Sparse autoencoders find composed features in small toy models. https://w.lesswrong.com/posts/a5wwqza2cY3W7L9cj/sparse-autoencoders-find-composed-features-in-small-toy, 03 2024. Accessed: 2025-08-11. Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. Conerly et al. [2024] Tom Conerly, Adly Templeton, Trenton Bricken, Jonathan Marcus, and Tom Henighan. Update on how we train saes. Transformer Circuits Thread, April 2024. URL https://transformer-circuits.pub/2024/april-update/index.html#training-saes. Part of Circuits Updates - April 2024, Anthropic Interpretability Team. Conerly et al. [2025] Tom Conerly, Hoagy Cunningham, Adly Templeton, Jack Lindsey, Basil Hosmer, and Adam Jermyn. Dictionary learning optimization techniques. Transformer Circuits Thread, January 2025. URL https://transformer-circuits.pub/2025/january-update/index.html#DL. Part of Circuits Updates - January 2025, Anthropic Interpretability Team. Appendix A Additional discussion A.1 Are real SAEs in the pathological regime? It is worth attempting to say more about whether SAEs are in the pathological scaling regime in practice. As we stated in Section 2.5, this depends on the rate at which the feature frequencies pi∝i−(1+α)p_i i^-(1+α) decay vs. the rate at which the per-feature SAE loss decays L(ni)∝i−βL(n_i) i^-β. In Appendix B, following Brill (2024) Brill [2024], we show that when α<βα<β, the efficiency at which SAEs discover features D(N)/ND(N)/N approaches a reasonable constant, but when β<αβ<α, D(N)/ND(N)/N approaches 0 as N→∞N→∞. Whether feature manifolds could cause pathological SAE scaling in the real world depends then on the real α (assuming it’s even a power law) and β (assuming Li(ni)L_i(n_i) is also a power law ∝ni−β n_i^-β). What is α?: We first speculate on what α may be. One way of trying to measure this is to look at how the latent activation frequencies decay when sorted by frequency. For a few Gemma Scope SAEs Lieberum et al. [2024], we show these curves in Figure 4, and measure slopes between −0.57-0.57 and −0.74-0.74. If there was a one-to-one relationship between SAE latents and features in the data, then this would imply an α≈0.5α≈ 0.5 to α≈0.7α≈ 0.7. However, feature absorption Chanin et al. [2024], the learning of compositional features Till [2024], Anders et al. [2024], and latents tiling a feature manifold could distort the relationship between the true feature frequencies and the latent activation frequencies. Figure 4: Frequencies at which latents fire in gemma scope SAEs, sorted by frequency. We measure the power law decay exponent between latent 10210^2 and 10410^4 to be -0.74 for an SAE with 16k latents and -0.57 for an SAE with 32k latents. Another highly speculative way of trying to estimate α could be to look at the exponents of neural scaling laws for models like those the SAE is being trained on. The idea here is that if the “features” are the computational units of neural networks that ref Michaud et al. [2023] called the “quanta”, then the underlying neural scaling law slope would reflect the distribution over feature occurrences. Neural scaling exponents for language models (w.r.t. network parameters) (αN _N in the scaling law N−αN^- _N have been measured to have an αN _N of 0.070.07 Kaplan et al. [2020] and 0.340.34 Hoffmann et al. [2022], potentially implying a distribution over quanta/features pi∝i−(1+α)p_i i^-(1+α) with exponent α in that same range. What is β?: In Figure 3 and Figure 6, we show L(n)L(n) scaling curves for SAEs reconstructing single synthetic feature manifolds. We find that these curves depend on the feature geometry. In particular, we see that when reconstructing hollow hyperspheres, we can observe gradual L(n)L(n) scaling. The higher-dimensional hyperspheres in particular can accommodate at least 10410^4 latents without loss plateauing. In Figure 5, we plot the slope that we measure for these curves between 10210^2 and 10410^4 latents, and measure a β of roughly 0.050.05 for hyperspheres with dimension 66-88. Figure 5: Measured slopes of LiL_i curves for ReLU L1 SAEs trained on hyperspheres. However, in the more realistic setting where there is variation in the radial direction–the intensity that features fire Olah [2024]–we see that manifolds tend to saturate very quickly. It appears that the saturation happens when ni≈2din_i≈ 2d_i, likely corresponding to solutions where the SAE latents form a basis for the subspace where the feature is embedded (or rather use two latents for each basis direction, one in the “positive” direction and one in the “negative” direction since latents can only fire positively). Therefore, the slope of the Li(ni)L_i(n_i) curve, and whether SAEs can use a large number of latents to reduce loss on feature manifolds, depends on the geometry of the manifold. Our experiments so far indicate that in the more realistic setting where there is variation in the radial direction (which was seen in practice in ref Engels et al. [2024a]), that SAEs do not discover solutions which take advantage of a large number of latents, and instead learn a basis solution. This is probably the strongest argument against the possibility of feature manifolds causing pathological SAE scaling. Lastly, we note that we are unsure how “ripples” Olah [2024] in feature manifolds could affect SAE scaling on them. If a feature manifold is intrinsically low dimensional, but ripples through a large number of other dimensions, we could imagine SAE solutions potentially looking different. Appendix B Derivations B.1 Loss decomposition into per-feature terms We work under the assumptions in §2.2: (i) feature-specific latents—each latent j fires only when a unique feature i(j)i(j) is active; (i) decoder respect for subspaces—decoder columns for latents assigned to feature i lie in span(i)span(S_i); and (i) orthogonal feature subspaces—span(i)⟂span(k)span(S_i) (S_k) for i≠ki≠ k. Additivity of sparsity. For a sample =∑i∈()iix= _i (x)S_if_i, feature-specificity implies ^j()=0 f_j(x)=0 unless i(j)∈()i(j) (x). For separable sparsity penalties (L0/L1), S(^)=∑js(f^j)S( f)= _js( f_j), so S(^())=∑i∈()∑j:i(j)=is(f^j()).S( f(x))= _i (x)\; _j:\,i(j)=is( f_j(x))\,. Thus the sparsity cost splits across active features. Orthogonal reconstruction. Write the model reconstruction as ^=∑i^i x= _i x_i, with ^i:=∑j:i(j)=ijf^j∈span(i) x_i := _j:\,i(j)=iw_j f_j (S_i) by (i). Then, using (i) and the Pythagorean theorem, ‖−^‖22=‖∑i(ii−^i)‖22=∑i‖ii−^i‖22.\|x- x\|_2^2= \| _i (S_if_i- x_i ) \|_2^2= _i\|S_if_i- x_i\|_2^2\,. Hence the per-sample objective ℒ()=‖−^‖22+λS(^)L(x)=\|x- x\|_2^2+λ S( f) decomposes as a sum over features active on that sample. Taking expectations and letting nin_i be the number of latents allocated to feature i, we obtain [ℒ()]=∑ipiLi(ni),∑ini=N _x [L(x) ]= _ip_i\,L_i(n_i), _in_i=N (1) where pi:=Pr[i≠0]p_i := [f_i≠ 0] and Li(ni)L_i(n_i) is the optimal (feature-i) expected reconstruction-plus-sparsity loss achieved with nin_i latents restricted to span(i)span(S_i). We note that the derivations below closely follow those in Brill (2024) Brill [2024], adapted to our SAE setting with appropriate changes in notation and interpretation. We show them here for convenience. B.2 Scaling setup and notation We study the optimal latent allocation nin_i minimizing (1) under two empirical power-law regularities: pi∝i−(1+α)(features sorted by frequency),Li(n)≡L(n)∝n−β,p_i i^-(1+α) (features sorted by frequency), L_i(n)≡ L(n) n^-β\,, with α,β>0α,β>0.111Constants are inessential for power-law exponents and are dropped. Define the discovery count D(N):=|i:ni>0|D(N) := |\\,i:\;n_i>0\,\ | and the total expected loss ℒ(N):=∑ipiL(ni)L(N) := _ip_iL(n_i) at total width N. A standard Lagrange multiplier treatment (continuous relaxation) yields ni∝pi11+β∝i−γ,γ:=1+α1+β.n_i\; \;p_i 11+β\; \;i^-γ, γ := 1+α1+β. (2) In practice there is a cutoff index ici_c (“last discovered feature”) with nic≈1n_i_c≈ 1 and ni≲1n_i 1 for i>ici>i_c. Then D(N)≍icD(N) i_c and N=∑i≤icni∝∑i≤ici−γ.N= _i≤ i_cn_i\; \; _i≤ i_ci^-γ. Two regimes follow depending on whether the allocation tail-sum diverges or converges. B.3 Case β<αβ<α (simple, latent accumulation on frequent features) Here γ>1γ>1, so ∑i≥1i−γ _i≥ 1i^-γ converges to a constant Z(γ)Z(γ). From (2), N∝∑i≤ici−γ→Z(γ)N _i≤ i_ci^-γ→ Z(γ) implies the proportionality constant in (2) scales as κ∝Nκ N. The discovery cutoff is set by nic≈1n_i_c≈ 1: 1≈κic−γ⟹ic∝κ1/γ∝N1γ=N1+β1+α.1\;≈\;κ\,i_c^-γ i_c\; \;κ^1/γ\; \;N 1γ=N 1+β1+α. Thus D(N)∝N1+β1+α(sublinear discovery). D(N)\; \;N 1+β1+α (sublinear discovery). (3) For the loss, the discovered part scales as ∑i≤icpini−β∝κ−β∑i≤ici−γ∝κ−β∝N−β, _i≤ i_cp_i\,n_i^-β\; \;κ^-β\! _i≤ i_ci^-γ\; \;κ^-β\; \;N^-β, while the undiscovered tail ∑i>icpi∝ic−α∝N−α(1+β)/(1+α) _i>i_cp_i i_c^-α N^-α(1+β)/(1+α) decays faster since α>βα>β. Therefore ℒ(N)∝N−β. (N)\; \;N^-β. (4) Intuitively, the SAE keeps shaving loss on common feature manifolds; discovery lags. B.4 Case α<βα<β (benign, feature discovery keeps up) Now γ<1γ<1, so ∑i≤ici−γ∝ic1−γ _i≤ i_ci^-γ i_c^1-γ. Using N∝κic1−γN κ\,i_c^1-γ and the threshold 1≈κic−γ1≈κ\,i_c^-γ, we eliminate κ to find N∝ic⟹D(N)∝N.N\; \;i_c \;D(N)\; \;N.\; For the loss over discovered features, ∑i≤icpini−β∝κ−β∑i≤ici−γ∝ic−βγic1−γ=ic 1−(1+α)=ic−α∝N−α. _i≤ i_cp_i\,n_i^-β\; \;κ^-β _i≤ i_ci^-γ\; \;i_c^-βγ\,i_c^1-γ\;=\;i_c^\,1-(1+α)\;=\;i_c^-α\; \;N^-α. The undiscovered tail obeys ∑i>icpi∝ic−α∝N−α _i>i_cp_i i_c^-α N^-α, so both pieces match and ℒ(N)∝N−α. (N)\; \;N^-α. (5) Here, extra width primarily buys new features rather than over-tiling old manifolds; loss scaling mirrors the frequency tail. Appendix C Additional Experimental Details For Figure 3 and Figure 6, we trained SAEs on synthetic feature manifolds. For these experiments, we trained for 12000 steps, with a batch size of 2048, and a learning rate of 10−310^-3 with the Adam optimizer. For the L1 penalty calculation, we use the trick of multiplying the decoder vector L2 norms by the latent activation Conerly et al. [2024]. Appendix D SAE scaling curves on synthetic manifolds In Figure 6, we show JumpReLU SAE scaling on individual feature manifolds like we did for ReLU SAEs in Figure 3. Figure 6: JumpReLU SAE scaling on individual toy feature manifolds, showing how L(n)L(n) curves depend on feature geometry. We train with the tanh loss from Conerly et al. [2025] with c=0.1c=0.1 and λs=1.0 _s=1.0. Left: JumpReLU SAE scaling on unit hyperspheres of varying dimension. Right: JumpReLU SAE scaling on points sampled in : 0.5<||<2\x\ :\ 0.5<|x|<2\. We see that, like with ReLU SAEs, that when there is variation in the radial direction between samples that our SAEs do not learn solutions which can continue to accommodate latents, and instead plateau after allocating roughly 2di2d_i latents to the feature manifold. Appendix E SAE feature geometry on LLMs and vision models E.1 JumpReLU Gemma Scope SAEs If a large number of latents “tile” a single low-dimensional feature manifold, then we would expect the decoder directions for those SAE latents to have neighbors with high cosine similarity. One can see this effect directly in Figure 1, where we see that the SAE decoder latents begin to be arranged quite tightly together along the manifold. In this section, we study whether SAEs on real neural network activations have large numbers of latents with very high cosine similarity to their nearest neighbor. We first study this in the Gemma Scope SAEs Lieberum et al. [2024]. In Figure 7, we plot the distribution over cosine similarities between decoder latent vectors and their nearest neighbor for Gemma Scope SAEs on layer 12 (residual stream) of gemma-2-2b. While the distribution is skewed substantially higher than one would expect if all latent decoder vectors were random (and thus approximately orthogonal), we do not overall see a very large fraction of latents with extremely high cosine similarity to their nearest neighbor. Figure 7: Distribution over cosine similarities between decoder vectors and their nearest neighbor. Intriguingly though, for some SAEs we do see a small uptick on the right side of this distribution, where between 10-100 latents have cosine similarity >0.97>0.97 with their nearest neighbor. When we investigated these latents in one SAE (width_262k, average_l0_121), we found that for each of these latents, their nearest neighbor was dead (across a dataset of over 250 million tokens). These latents are not then being used to very sparsely reconstruct points on a manifold, and instead seem to be an artifact of the training process. However, the alive latents in this set are not typical SAE latents. A large number of these latents fire on tokens representing single numerical digits and single alphabet characters. We do not have an explanation of this phenomenon, but wonder whether there may be some underlying manifold representation which the SAE at one point in training tried to “tile”, but then when the latents got too close, one of them was killed to reduce the L0 loss. E.2 ReLU L1 SAEs on Inception-v1 We also study the geometry of latent decoder directions on SAEs trained on Inception-v1 activations. We train SAEs on activations from mixed3b using an L1 coefficient, λ, of 1, a learning rate of 10−410^-4, and an expansion factor of 16 (a total of 7680 latents). In Figure 8, we find that on Inception-v1, a meaningful fraction of SAE latents have very high cosine similarity with their nearest neighbor. We note however that this could be due to latents being duplicated, which is not strongly disincentivized by the L1 loss, as pointed out in Minder et al. [2025]. Figure 8: Distribution over pairwise cosine similarities for Inception V1.