Paper deep dive

The Origins of Representation Manifolds in Large Language Models

Alexander Modell, Patrick Rubin-Delanchy, Nick Whiteley

Year: 2025Venue: arXiv preprintArea: Representation AnalysisType: TheoreticalEmbeddings: 102

Models: GPT-2 Small, Mistral 7B, OpenAI text-embedding-large-3

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 3/12/2026, 6:48:00 PM

Summary

The paper proposes a 'continuous correspondence hypothesis' to explain how large language models represent complex, multidimensional features as manifolds rather than just atomic vectors. It argues that neural representations are homeomorphic to the intrinsic geometry of the concepts they encode, and demonstrates that cosine similarity in representation space effectively captures this intrinsic geometry through shortest paths on these manifolds.

Entities (5)

Linear Representation Hypothesis · theory · 98%Continuous Correspondence Hypothesis · hypothesis · 95%Sparse Autoencoders · methodology · 95%Representation Manifold · concept · 92%GPT-2 · model · 90%

Relation Signals (3)

Cosine Similarity → encodes → Intrinsic Geometry

confidence 95% · cosine similarity in representation space may encode the intrinsic geometry of a feature through shortest, on-manifold paths

Representation Manifold → ishomeomorphicto → Metric Space

confidence 95% · Proposition 1. Under Hypothesis 1, the map ϕf:Zf→Mf is a homeomorphism.

Sparse Autoencoders → recovers → Features

confidence 90% · This model underpins the use of sparse autoencoders to recover features from representations.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:There is a large ongoing scientific effort in mechanistic interpretability to map embeddings and internal representations of AI systems into human-understandable concepts. A key element of this effort is the linear representation hypothesis, which posits that neural representations are sparse linear combinations of `almost-orthogonal' direction vectors, reflecting the presence or absence of different features. This model underpins the use of sparse autoencoders to recover features from representations. Moving towards a fuller model of features, in which neural representations could encode not just the presence but also a potentially continuous and multidimensional value for a feature, has been a subject of intense recent discourse. We describe why and how a feature might be represented as a manifold, demonstrating in particular that cosine similarity in representation space may encode the intrinsic geometry of a feature through shortest, on-manifold paths, potentially answering the question of how distance in representation space and relatedness in concept space could be connected. The critical assumptions and predictions of the theory are validated on text embeddings and token activations of large language models.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

101,516 characters extracted from source content.

Expand or collapse full text

The Origins of Representation Manifolds in Large Language Models Alexander Modell Department of Mathematics Imperial College London a.modell@imperial.ac.uk &Patrick Rubin-Delanchy School of Mathematics University of Edinburgh prd@ed.ac.uk &Nick Whiteley School of Mathematics University of Bristol nick.whiteley@bristol.ac.uk Abstract There is a large ongoing scientific effort in mechanistic interpretability to map embeddings and internal representations of AI systems into human-understandable concepts. A key element of this effort is the linear representation hypothesis, which posits that neural representations are sparse linear combinations of ‘almost-orthogonal’ direction vectors, reflecting the presence or absence of different features. This model underpins the use of sparse autoencoders to recover features from representations. Moving towards a fuller model of features, in which neural representations could encode not just the presence but also a potentially continuous and multidimensional value for a feature, has been a subject of intense recent discourse. We describe why and how a feature might be represented as a manifold, demonstrating in particular that cosine similarity in representation space may encode the intrinsic geometry of a feature through shortest, on-manifold paths, potentially answering the question of how distance in representation space and relatedness in concept space could be connected. The critical assumptions and predictions of the theory are validated on text embeddings and token activations of large language models. 1 Introduction There is a large, ongoing, scientific effort in mechanistic interpretability to map internal representations used by AI systems into human-understandable concepts (Lin, 2024; Templeton et al., 2024), with broad implications for humanity including safety, alignment, and the future role of AI in science (Bostrom, 2014; Soares and Fallenstein, 2017; Wang et al., 2023). A key element of this effort is the linear representation hypothesis (LRH), which posits that language models represent human-interpretable features as directions in representation space, and that model representations are (literally) a sparse linear combination of these directions (Smolensky, 1990; Arora et al., 2018; Elhage et al., 2022). The methodology of sparse autoencoders (SAEs) (Elhage et al., 2022; Bricken et al., 2023) employs ideas from sparse coding (Elad, 2010) to estimate a dictionary of these directions from representations. This model and methodology reflect a radical goal of breaking representations down into basic, irreducible, atomic concepts which are meaningfully only described as present or absent (Cunningham et al., 2023; Bricken et al., 2023; Templeton et al., 2024). Commonly cited examples are features such as floppy_ears, Eiffel_Tower, or is_Arabic, the presence of which it would presumably be useful for an algorithm to infer (corresponding e.g. to cat/dog classification, the topic of a question, the language of a query). It is generally accepted that this breakdown of representation space into purely atomic features does not tell the whole story (Smith, 2024; Mendel, 2024; Bussmann et al., 2024; Olah, 2024; Engels et al., 2025). There is overwhelming empirical evidence that neural networks represent complex features in structures which unfold across multiple directions in potentially continuous, nonlinear ways: examples of curves (Hanna et al., 2023; Chang et al., 2022), swiss-roll-like manifolds (Cai et al., 2021), loops (Engels et al., 2025; Gorton, 2024), tori (Chang et al., 2022), hierarchical trees (Park et al., 2024) in real language models; topologically circular representations of numbers in toy models trained to perform modular arithmetic (Liu et al., 2022; Nanda et al., 2023a; Zhong et al., 2023; He et al., 2024) or simulated angular data (Olah and Batson, 2024), fractal geometry in simulated hidden Markov models (Shai et al., 2024); and broader phenomenology from local finite-state-automata (Bricken et al., 2023), to spatial ‘brain-like’ modularity (Li et al., 2025), to behaviour, such as deception (Templeton et al., 2024). SAEs are not made defunct by these discoveries, and in fact have often facilitated them through recombination of SAE directions (Bussmann et al., 2025; Engels et al., 2025). The LRH has been extended to allow this more flexible interpretation of the output of SAEs: Definition 1 (Multidimensional linear representation hypothesis). There exists a collection of features labeled ∈ f∈ Ftypewriter_f ∈ typewriter_F and associated subspaces V∈ℝDsubscriptsuperscriptℝV_ f ^DVtypewriter_f ∈ blackboard_RD such that the functional relationship between an input x∈x ∈ X and its representation Ψ⁢(x)Ψ (x)Ψ ( x ) is Ψ⁢(x)=∑∈⁢(x)ρ⁢(x)⁢v⁢(x),v⁢(x)∈V⁢ and ⁢‖v⁢(x)‖2=1,formulae-sequenceΨsubscriptsubscriptsubscriptsubscriptsubscript and subscriptnormsubscript21 (x)= _ f∈ F(x) _ f(x)v_ f(x)% , v_ f(x)∈ V_ f and \|v_ f(x)\|_% 2=1,Ψ ( x ) = ∑typewriter_f ∈ typewriter_F ( x ) ρtypewriter_f ( x ) vtypewriter_f ( x ) , vtypewriter_f ( x ) ∈ Vtypewriter_f and ∥ vtypewriter_f ( x ) ∥2 = 1 , (1) where ρ⁢(x)subscript _ f(x)ρtypewriter_f ( x ) is a non-negative scaling denoting the presence of the feature ftypewriter_f in x, and ⁢(x)=:ρ⁢(x)>0conditional-setsubscript0 F(x)=\ f\>:\> _ f(x)>0\typewriter_F ( x ) = typewriter_f : ρtypewriter_f ( x ) > 0 is the set of features which are present in x. The standard LRH corresponds to the case where v⁢(x)subscriptv_ f(x)vtypewriter_f ( x ) is constant in x (and VsubscriptV_ fVtypewriter_f one-dimensional), and the extension above is a slightly relaxed and reparametrised version of that which appears in Engels et al. (2025). Our paper concerns the representation of a feature ftypewriter_f as a manifold in VsubscriptV_ fVtypewriter_f, a phenomenon which is widely observed and intensely deliberated in the mechanistic interpretability community (Olah and Batson, 2024; Olah, 2024; Gorton, 2024; Engels et al., 2025). Despite numerous accounts (cited above) of a manifold clearly corresponding to some underlying ground truth feature (which may even be known exactly, e.g. in simulated data), there is no general description of this correspondence. We provide what we believe is a minimum viable mathematical theory to do this. Our most substantial, novel result establishes that under plausible hypotheses, cosine similarity in representation space encodes the intrinsic geometry of a feature through shortest, on-manifold paths. We develop this insight using concepts from metric geometry – the theory of length and shape in metric spaces (Burago et al., 2001). The widespread use of cosine similarity across data science could suggest many other applications for this result. More generally, our work provides a (hopefully) accessible explanation of why manifold structure might arise in representation space, how its topology and geometry might reflect a human conceptualisation of the feature, and suggests some simple diagnostic plots and statistical checks to explore critical assumptions and predictions of the theory. Related to the problem of mechanistic interpretability, there is enormous interest in the use of text embeddings (Li et al., 2020; Gao et al., 2021), which several entities now provide as a service, to support, for example, receiver augmented generation, search, recommendation, visualisation and classification. Again, these are representations which are usually unit vectors, and about which the cosine similarity is said to provide an effective measure of semantic similarity. The bulk of our results are also applicable to this area; in what follows, Ψ⁢(x)Ψ (x)Ψ ( x ) can be viewed as a generic representation of some input x, and v⁢(x)subscriptv_ f(x)vtypewriter_f ( x ) some unit-norm representation of a feature in x. 2 The continuous correspondence hypothesis 2.1 What is a feature anyway? Before we begin our discussion on how features are geometrically represented in language models, we really ought to pin down what exactly we mean by “a feature”. Definition 2. A feature, labeled ftypewriter_f, is a metric space (,)subscriptsubscript(Z_ f, d_ f)( Ztypewriter_f , sansserif_dtypewriter_f ). A metric space is simply a set equipped with a distance, and we find that it provides a simple yet highly expressive formal mathematical framework for discussing the abstract notion of a feature or concept. In particular, it allows us to readily talk about: 1. Atomic features: subscriptZ_ fZtypewriter_f a singleton set. 2. Hierarchical features: subscriptZ_ fZtypewriter_f a discrete set, subscript d_ fsansserif_dtypewriter_f a tree distance. 3. Continuous features: subscriptZ_ fZtypewriter_f an interval (equipped with e.g. (x,y)=|x−y|) d_ f(x,y)=|x-y|)sansserif_dtypewriter_f ( x , y ) = | x - y | ), a circle (equipped with e.g. arc-length distance), multi-dimensional (equipped with e.g. the Euclidean distance), etc. Figure 1: Representation manifolds in large language models: colours, years and dates. The first and third example show text embeddings obtained from OpenAI’s text-embedding-large-3 model from prompts relating to English names for colours and dates of the year, respectivly. The second example shows token activations from layer 7 of GPT2-small, which were studied in Engels et al. (2025). The token activations were processed via an SAE to extract a feature corresponding to years of the twentieth century as in Engels et al. (2025), and normalized to have norm one. For each example, we perform principal component analysis (PCA) to reduce the dimension to three and display the resulting point clouds from two perspectives. The embeddings of English names for colours are displayed in their respective colour value. Years are coloured from blue (1900) through green to yellow (1999), and dates are coloured from white (1st Janurary) through blue to black (1st July) through red and back to white. We find that this formalism strikes a balance between the less expressive Euclidean and hyperspherical models often assumed in the learning theory literature (e.g. Zimmermann et al., 2021; Hyvärinen et al., 2024; Reizinger et al., 2025), and the more complicated and less accessible models which are often assumed in the disentanglement literature, such as Riemannian manifolds equipped with group structure (e.g. Higgins et al., 2018; Pfau et al., 2020). For any input x on which the feature ftypewriter_f is present (i.e. for which ρn⁢(x)>0subscript0 _n(x)>0ρitalic_n ( x ) > 0), we assume the existence of a value z⁢(x)subscriptz_ f(x)ztypewriter_f ( x ) which the input takes in subscriptZ_ fZtypewriter_f. For example, if the feature colour is present in an input x, then ρ⁢(x)>0subscript0 _ colour(x)>0ρtypewriter_colour ( x ) > 0 and z⁢(x)subscriptz_ f(x)ztypewriter_f ( x ) might take a value describing the precise hue, saturation and lightness of that colour. As a final note, we will assume throughout this paper that each subscriptZ_ fZtypewriter_f is a compact set or, loosely speaking, “closed and bounded”: a standard assumption in manifold learning which avoids considerable and possibly distracting theoretical complications. 2.2 The continuous correspondence hypothesis Given the multidimensional linear representation hypothesis, and our definition of a feature, perhaps the most basic hypothesis that one can make is that there is some way of matching the representation directions v⁢(x)subscriptv_ f(x)vtypewriter_f ( x ) to the abstract features z⁢(x)subscriptz_ f(x)ztypewriter_f ( x ). Hypothesis 1 (continuous correspondence). The features z⁢(x)subscriptz_ f(x)ztypewriter_f ( x ) and representation directions v⁢(x)subscriptv_ f(x)vtypewriter_f ( x ) are in a continuous, one-to-one correspondence. Formally, there is a continuous invertible map from the metric space into the hypersphere, ϕ:→D−1:subscriptitalic-ϕ→subscriptsuperscript1 _ f:Z_ f ^D-1ϕtypewriter_f : Ztypewriter_f → blackboard_SD - 1, with image ℳ≔ϕ⁢()≔subscriptℳitalic-ϕM_ f φ(Z)Mtypewriter_f ≔ ϕ ( Z ), such that v⁢(x)=ϕ⁢(z⁢(x))subscriptsubscriptitalic-ϕsubscriptv_ f(x)= _ f(z_ f(x))vtypewriter_f ( x ) = ϕtypewriter_f ( ztypewriter_f ( x ) ) for all x∈x ∈ X. Our correspondence hypothesis, combined with the prior assumption that subscriptZ_ fZtypewriter_f is compact, has an immediate implication for the topological relationship between subscriptZ_ fZtypewriter_f and ℳsubscriptℳM_ fMtypewriter_f. Proposition 1. Under Hypothesis 1, the map ϕ:→ℳ:subscriptitalic-ϕ→subscriptsubscriptℳ _ f:Z_ f _ fϕtypewriter_f : Ztypewriter_f → Mtypewriter_f is a homeomorphism.111This is simply a restatement of the well-established fact that a continuous invertible map over a compact domain has a continuous inverse (Sutherland, 2009, Proposition 13.26). Proposition 1 tells us that under the continuous correspondence hypothesis, we should expect the representations directions v⁢(x)subscriptv_ f(x)vtypewriter_f ( x ) to live on a manifold ℳsubscriptℳM_ fMtypewriter_f that is topologically identical to subscriptZ_ fZtypewriter_f. So, if subscriptZ_ fZtypewriter_f is an interval, ℳsubscriptℳM_ fMtypewriter_f is a one-dimensional curve in ℝDsuperscriptℝR^Dblackboard_RD. If subscriptZ_ fZtypewriter_f is a circle, then ℳsubscriptℳM_ fMtypewriter_f is a loop. If subscriptZ_ fZtypewriter_f is a discrete set comprising m values, ℳsubscriptℳM_ fMtypewriter_f is a discrete set comprising m points. More generally, a homeomorphism preserves connected components, holes, branching points, and more. Figure 2: Representation manifolds in token activations from layer 8 of Mistral 7B, processed via an SAE to extract representations of ‘months of the year’ and ‘days of the week’, as in Engels et al. (2025). We normalise the representations to have norm one, and perform PCA into three dimensions. The top-down view of the first two principal components, which was shown in Engels et al. (2025), obscures manifold structure which weaves through the third principal component. 2.3 Representations reflect the topology of features in LLMs Figure 1 gives an indiction of the plausibility of Hypothesis 1 in some examples. The first and third subfigures show text embeddings obtained from OpenAI’s text-embedding-large-3 model, with inputs corresponding to colours222These inputs are of are of the form “The color of the object is ¡color¿. What color is the object?”. Color names and hex-codes were obtained from the XKCD color survey (Munroe, 2010), from which we removed entries with low saturation (<0.4absent0.4<0.4< 0.4), high brightness (>0.8absent0.8>0.8> 0.8), and whose names did not obviously refer to a color, such as fruits and gemstones. Some additional outliers were removed., and dates of the year333These inputs are of the form “1st January”, ”2nd January”, … “31st December”. respectively. This model returns 3,072 dimensional unit-norm embeddings which we reduce to three dimensions using PCA. We show two perspectives of each plot. In both cases, we see that the embeddings are roughly arranged around a loop which, perhaps after some stretching and bending, could seem consistent with the abstract circular model we might have for such concepts, such as the “colour wheel” or the “yearly cycle”. In particular, observe that the colours are arranged in the same order as the standard colour wheel of hue: red, purple, blue, green, yellow, orange, and back to red. The second subfigure shows token activations of years of the twentieth century in layer 7 of GPT2-small. This example is taken from Engels et al. (2025) who use a sparse autoencoder to attempt to disentangle the feature representations from the full superposed representation (see Section 5 of their paper for addition details of this procedure). We subselect only tokens corresponding to the years in question, normalize each activation vector to have norm one and perform PCA into three dimensions on the resulting vectors. One observes a clear one-dimensional curve which weaves and bends through the dimensions of the space, again reminiscent up to some geometric distortion of the standard human concept of a “time line”. Given what we see, our innate understanding of these concepts, and Proposition 1, we might conjecture that, allowing for error of different kinds, the shapes are homeomorphic to the following metric spaces: colour: colour=[0,2⁢π)subscriptcolour02Z_ colour=[0,2π)Zcolour = [ 0 , 2 π ), colour⁢(x,y)=min⁡(|x−y|,2⁢π−|x−y|)subscriptcolour2 d_ colour(x,y)= (|x-y|,2π-|x-y|)sansserif_dcolour ( x , y ) = min ( | x - y | , 2 π - | x - y | ), these angles corresponding to hues 0: red, …, π/33π/3π / 3: blue, …, 2⁢π/3232π/32 π / 3: yellow. years: years=[1900,1999]subscriptyears19001999Z_ years=[1900,1999]Zyears = [ 1900 , 1999 ], year⁢(x,y)=|x−y|subscriptyear d_ year(x,y)=|x-y|sansserif_dyear ( x , y ) = | x - y | dates: dates=[0,365)subscriptdates0365Z_ dates=[0,365)Zdates = [ 0 , 365 ), dates⁢(x,y)=min⁡(|x−y|,365−|x−y|)subscriptdates365 d_ dates(x,y)= (|x-y|,365-|x-y|)sansserif_ddates ( x , y ) = min ( | x - y | , 365 - | x - y | ) In the case of years, a simple statistic to assess the conjecture of homeomorphism presents itself: the rank correlation between the years and their corresponding position along the manifold. We approximate position along the manifold using a K-nearest neighbour graph with K=1010K=10K = 10 (picked as small as possible subject to the graph being connected), and rank the points according to weighted graph distance from the (mean) representation of 1900. The Kendall and Spearman rank correlations are 0.97 and over 0.99, respectively, telling us that the representations occur in very close to true temporal order along the manifold. In these examples it would clearly not be reasonable to say the shapes resembled circles or straight lines without any sort of geometric distortion, and in the coming section we provide a mechanistic argument for the presence of this geometric distortion in neural networks. This effect could be missed in some earlier papers due to 2D projection. The left and middle of panels of Figure 1 of Engels et al. (2025) show circular arrangements of day-of-the-week and month representations, but these seem subject to significant geometric distortion once we view the data in 3D, as in Figure 2. 2.4 Manifold geometry and computation Our investigations (see Figure 1) and those of many others (see e.g. Ansuini et al., 2019; Cai et al., 2021; Chang et al., 2022; Hanna et al., 2023), have found not only that representations tend to live on low-dimensional manifolds, but that these manifolds curve and bend to occupy higher dimensional spaces. Why might it be advantageous for a language model to embed a concept in a larger dimension than its intrinsic topology seems to require? To answer this question, we shall briefly illustrate how the space of functions which can be computed as a linear projection of ϕ⁢(z)subscriptitalic-ϕ _ f(z)ϕtypewriter_f ( z ) relates to its geometry. Since linear operations are crucial component in how one layer of a neural network maps to the next, it seems a sensible working hypothesis that they would arrange their representations as to maximize the expressivity of these linear computations. For the purpose of this discussion, consider the case that subscriptZ_ fZtypewriter_f is a unit interval =[0,1]subscript01Z_ f=[0,1]Ztypewriter_f = [ 0 , 1 ]. If one simply wanted to be able to read z from ϕ⁢(z)subscriptitalic-ϕ _ f(z)ϕtypewriter_f ( z ) using a linear projection, then it is sufficient to represent subscriptZ_ fZtypewriter_f as an arc on D−1superscript1S^D-1blackboard_SD - 1. For example, to set ϕ⁢(z)=b0⁢(z)⁢v0+b1⁢(z)⁢v1subscriptitalic-ϕsubscript0subscript0subscript1subscript1 _ f(z)=b_0(z)v_0+b_1(z)v_1ϕtypewriter_f ( z ) = b0 ( z ) v0 + b1 ( z ) v1 where v0,v1∈D−1subscript0subscript1superscript1v_0,v_1 ^D-1v0 , v1 ∈ blackboard_SD - 1 are orthogonal unit-vectors, b1⁢(z)∝zproportional-tosubscript1b_1(z) zb1 ( z ) ∝ z, and b0⁢(z)subscript0b_0(z)b0 ( z ) is a function which ensures that ‖ϕ⁢(z)‖2=1subscriptnormsubscriptitalic-ϕ21\| _ f(z)\|_2=1∥ ϕtypewriter_f ( z ) ∥2 = 1. In this way, the identity operation id⁡(z):=zassignidid(z):=zid ( z ) := z can be computed via a linear projection id⁡(z)∝v1⋅ϕ⁢(z)proportional-toid⋅subscript1subscriptitalic-ϕid(z) v_1· _ f(z)id ( z ) ∝ v1 ⋅ ϕtypewriter_f ( z ). If instead, one wanted to be able to represent a richer class of functions of z by linear projections of ϕ⁢(z)subscriptitalic-ϕ _ f(z)ϕtypewriter_f ( z ), say, polynomials of order p, then one could do this by setting ϕ⁢(z)=b0⁢(z)⁢v0+⋯⁢bp+1⁢(z)⁢vp+1subscriptitalic-ϕsubscript0subscript0⋯subscript1subscript1 _ f(z)=b_0(z)v_0+·s b_p+1(z)v_p+1ϕtypewriter_f ( z ) = b0 ( z ) v0 + ⋯ bitalic_p + 1 ( z ) vitalic_p + 1 with b1⁢(z)∝1,b2⁢(z)∝z,b3∝z2,formulae-sequenceproportional-tosubscript11formulae-sequenceproportional-tosubscript2proportional-tosubscript3superscript2b_1(z) 1,b_2(z) z,b_3 z^2,b1 ( z ) ∝ 1 , b2 ( z ) ∝ z , b3 ∝ z2 , etc… Such a map represents the interval [0,1]01[0,1][ 0 , 1 ] as a continuous path which weaves through a p+22p+2p + 2-dimensional subspace of D−1superscript1S^D-1blackboard_SD - 1. Superposition. Under the Linear Representation Hypothesis (1), the language model cannot access ϕ⁢(z⁢(x))subscriptitalic-ϕsubscript _ f(z_ f(x))ϕtypewriter_f ( ztypewriter_f ( x ) ) directly, but must do so via Ψ⁢(x)Ψ (x)Ψ ( x ). There is a generally agreed upon explanation, known as the superposition hypothesis, for how an algorithm might nonetheless be granted approximate access to ϕ⁢(z⁢(x))subscriptitalic-ϕsubscript _ f(z_ f(x))ϕtypewriter_f ( ztypewriter_f ( x ) ), with only limited interference from other ϕ′⁢(z′⁢(x))subscriptitalic-ϕsuperscript′subscriptsuperscript′ _ f (z_ f (x))ϕtypewriter_f′ ( ztypewriter_f′ ( x ) ): features occur only sparsely (i.e. ρ⁢(x)=0subscript0 _ f(x)=0ρtypewriter_f ( x ) = 0 for most ∈ f∈ Ftypewriter_f ∈ typewriter_F), and are represented in almost-orthogonal subspaces (Elhage et al., 2021, 2022), a hypothesis which, in particular, would be consistent with the total number of features being substantially greater than the available representation dimensions444see, for example, Theorem 1 in the appendix of Engels et al. (2025).. If we assume (for a real-valued feature) that the identity id⁡(z)idid(z)id ( z ) is among the collection of functions linearly readable from ϕsubscriptitalic-ϕ _ fϕtypewriter_f, the superposition hypothesis also explains the efficacy of linear probes (Alain and Bengio, 2017; Gurnee and Tegmark, 2023; Nanda et al., 2023b; Leask et al., 2024): low interference allows the feature of interest to be approximately recovered from the representation using linear regression. A similar story holds for discrete features accessed via linear classifiers. 3 The interpretation of distance on representation manifolds There is an open question in the mechanistic interpretability community about the meaning of distance in representation space, perhaps well-summarised in the commentary of Olah and Batson (2024): “We suspect this idea that feature manifolds many be embedded in more complex ways than their topology suggests, in order to achieve a given distance metric, may actually be quite deep and important.” Figure 3: Evidence for Hypothesis 2 and its implications in Theorem 1. For each pair of representations, we plot their cosine similarities (first row) and estimated manifold distances (second row) against their (squared) distance in a putative metric space. We report the Chatterjee (ξ) and Pearson (ρ) correlation coefficients, respectively. Colours correspond to the colourmaps described in Figure 1. If we accept there is a correspondence between features and their representations (Hypothesis 1), arguably the next most basic question we can ask is whether cosine similarity in representation space somehow tells us about distance between the corresponding feature values. Hypothesis 2 (cosine similarity reflects distance). Locally, the cosine similarity between feature representations and the distance their corresponding feature values are inversely related. Formally, there is some function gsubscriptg_ fgtypewriter_f with continuous second derivatives and with g′⁢(0)<0superscriptsubscript′00g_ f (0)<0gtypewriter_f′ ( 0 ) < 0, and some ϵ>0italic-ϵ0ε>0ϵ > 0, such that ⁡(ϕ⁢(z),ϕ⁢(z′))=g⁢(⁢(z,z′)2),subscriptitalic-ϕsubscriptitalic-ϕsuperscript′subscriptsubscriptsuperscriptsuperscript′2 CosSim ( _ f(z), _ % f(z ) )=g_ f( d_ f(z,z )^2% ),sansserif_CosSim ( ϕtypewriter_f ( z ) , ϕtypewriter_f ( z′ ) ) = gtypewriter_f ( sansserif_dtypewriter_f ( z , z′ )2 ) , for all z,z′∈superscript′subscriptz,z _ fz , z′ ∈ Ztypewriter_f such that ⁢(z,z′)≤ϵsubscriptsuperscript′italic-ϵ d_ f(z,z )≤ _dtypewriter_f ( z , z′ ) ≤ ϵ. Strengthening just Hypothesis 1 to both Hypotheses 1 and 2 has formidable consequences: there is an intrinsic sense in which a feature and its representation are geometrically indistinguishable. This statement is made precise in Theorem 1. Metric spaces allow for a natural definition of a path which, loosely speaking, captures the idea of a continuous route from one point to another and there is an associated definition of the length of a path, denoted L, which generalises the usual Euclidean notion of length (Burago et al., 2001). Formally, a path in subscriptZ_ fZtypewriter_f is a continuous mapping η from some interval [a,b][a,b][ a , b ] to subscriptZ_ fZtypewriter_f, and the length of such a path is L⁢(η)≔sup∑i=1n⁢(ηti,ηti−1),≔subscriptsupremumsuperscriptsubscript1subscriptsubscriptsubscriptsubscriptsubscript1L(η) _T _i=1^n d_ f( _% t_i, _t_i-1),L ( η ) ≔ supcaligraphic_T ∑i = 1n sansserif_dtypewriter_f ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) , where the supremum is over all n≥11n≥ 1n ≥ 1 and =(t0,t1,…,tn)subscript0subscript1…subscriptT=(t_0,t_1,…,t_n)T = ( t0 , t1 , … , titalic_n ) such that t0=a≤t1≤⋯≤tn=bsubscript0subscript1⋯subscriptt_0=a≤ t_1≤·s≤ t_n=bt0 = a ≤ t1 ≤ ⋯ ≤ titalic_n = b. Given a path in subscriptZ_ fZtypewriter_f, we can think of the image of this path when mapped through ϕsubscriptitalic-ϕ _ fϕtypewriter_f, which we call the corresponding path on ℳsubscriptℳM_ fMtypewriter_f. Its length is defined similarly, with Euclidean distance in place of subscript d_ fsansserif_dtypewriter_f. Theorem 1. Let η be a path on subscriptZ_ fZtypewriter_f of finite length and, assuming Hypothesis 1, let γ be the corresponding path on ℳsubscriptℳM_ fMtypewriter_f. Then, under Hypothesis 2, L⁢(γ)=−2⁢g′⁢(0)⁢L⁢(η).2superscriptsubscript′0L(γ)= -2g_ f (0)L(η).L ( γ ) = square-root start_ARG - 2 gtypewriter_f′ ( 0 ) end_ARG L ( η ) . A proof of Theorem 1 is given in the appendix. Theorem 1 tells us that we can recover the intrinsic geometry of subscriptZ_ fZtypewriter_f, even though we know (almost) nothing about gsubscriptg_ fgtypewriter_f: shortest paths on ℳsubscriptℳM_ fMtypewriter_f correspond to shortest paths on subscriptZ_ fZtypewriter_f, and their lengths are equal, up to a choice of unit (reflected by −2⁢g′⁢(0)2superscriptsubscript′0 -2g_ f (0)square-root start_ARG - 2 gtypewriter_f′ ( 0 ) end_ARG) Figure 4: Evidence against isometry with respect to the metric space years=[1900,1999]subscriptyears19001999Z_ years=[1900,1999]Zyears = [ 1900 , 1999 ], year⁢(x,y)=|x−y|subscriptyear d_ year(x,y)=|x-y|sansserif_dyear ( x , y ) = | x - y |. There is no clear regular linear relationship between distances in this metric space and estimated distances on the representation manifold. The colours indicate that distances between more recent years are expanded on the manifold. 3.1 Geodesic distances on representation manifolds of LLMs are meaningful We now explore the plausibility of Hypothesis 2 in the same colour, year, date examples considered in Section 2.3 and Figure 1. In all cases, we find indications of isometry, with some important caveats. Given a putative metric space, Hypothesis 2 suggests two diagnostic tests. The first (direct) approach is to plot cosine similarity against squared distance, to check if the first appears to be a decreasing function of the second, around zero, up to noise. We quantify the global strength of functional dependence using Chatterjee’s correlation coefficient ξ (Chatterjee, 2021), which would be 1 if the cosine similarity was a deterministic function of distance. These experiments are shown in the first row of Figure 3. The second (indirect) approach is to test Theorem 1: geodesic distance on ℳsubscriptℳM_ fMtypewriter_f (shortest path length) should be linear in the geodesic distance on subscriptZ_ fZtypewriter_f, up to noise (the slope being −2⁢g′⁢(0)2superscript′0 -2g (0)square-root start_ARG - 2 g′ ( 0 ) end_ARG). We estimate geodesics on ℳsubscriptℳM_ fMtypewriter_f by constructing the K-nearest-neighbours graph over the representations, and reporting weighted graph distance, k chosen as small as possible subject to the graph being connected. We quantify the strength of isometry using Pearson’s correlation ρ, which would be 1 if the distances were in a perfect proportional relationship. These experiments are shown in the second row of Figure 3. Across our experiments, we have found that a low-dimensional projection tends to be necessary for the representations to plausibly show isometry with a simple metric space. For our text embeddings, we find that projecting onto the first few (uncentered) principal components works well. The routine “low-rank” explanation that the remaining components are mostly noise seems disputable; these components often show clear structure. Our best explanation is that semantic similarity is much richer than the rudimentary metric spaces to which they are being compared. It is likely that we could achieve a deeper understanding of semantic similarity through improved metric space design. In the example of years, the process of extracting feature representation via an SAE automatically yields low-dimensional representations, so no PCA is applied in this case. Recall that we conjectured the following metric space for the years example: years=[1900,1999]subscriptyears19001999Z_ years=[1900,1999]Zyears = [ 1900 , 1999 ], year⁢(x,y)=|x−y|subscriptyear d_ year(x,y)=|x-y|sansserif_dyear ( x , y ) = | x - y |. Although we found a rank correlation near 1, indicating homeomorphism, the evidence of the tests above is against isometry. The clearest indication in this direction is possibly provided by the right panel of Figure 4, which does not show a regular linear relationship, the colours suggesting that distances between more recent years are expanded on the manifold. In light of this, we consider a modified representation years=log⁡(2019−year):year∈[1900,1999]subscriptyearsconditional-set2019yearyear19001999Z_ years=\ (2019-year):year∈[1900,1999]\Zyears = log ( 2019 - year ) : year ∈ [ 1900 , 1999 ] , year⁢(x,y)=|x−y|subscriptyear d_ year(x,y)=|x-y|sansserif_dyear ( x , y ) = | x - y |, 2019 being the year GPT-2 was released (Radford et al., 2019). Observe that the rank correlation as computed in Section 2.3 remains unchanged: the two representations are homeomorphic, and cannot be distinguished on purely topological criteria. The tests are now in much stronger support of isometry. The top-middle plot of Figure 3 shows a trend which is clearly decreasing at zero, and a globally high functional dependence, 0.84. The bottom-middle panel of Figure 3 shows a clear linear fit, achieving a Pearson correlation of 0.99. Isometry is also found to be plausible for dates, and weakly plausible for colours, both with the original circular metric spaces conjectured in Section 2.3. The bottom-right panel shows a clear linear fit, achieving a Pearson correlation of 0.97. Observe in this case that the cosine similarity appears to be inconsistent with the metric for large distances, illustrating the point that Hypothesis 2 only requires an inverse functional relationship to hold locally. 4 Discussion, limitations and future work This work provides a formal mathematical framework to explain and interpret representation manifolds in large language models. By modeling features as metric spaces, we are able to accurately characterise the topological and geometric properties of their representations under some basic hypotheses. We perform some preliminary investigations on internal representations from GPT2-small and text embeddings from OpenAI’s text-embedding-large-3 model, which validate our theory and provide a nuanced, quantitative view of the correspondence between human concepts and their representation at different levels of geometric fidelity, namely homeomorphism and isometry. We find hints that these models encode distances in ways which are sometimes unexpected: years of the twentieth century in GPT2-small appear to be encoded on a logarithmic scale, with larger distances between more recent years, and colours in text-embedding-large-3 appear to be encoded in a cycle of hues, rather than representations that other systems might have chosen, such as RGB or wavelength. 4.1 Limitations In the spirit of scientific investigation, we have opted for a hypothesis-driven approach to structure discovery, in which we put down a possible metric space as a hypothesis and then assess evidence in favour or against. This “manual” approach is clearly not scalable, and moreover relies on there being some reasonable starting hypothesis for the metric space, which could be difficult for many features (say, emotions), as is evident from prior research (Li et al., 2023; Nanda et al., 2023b). There is an unexplored alternative approach of learning the metric, but we do not know exactly how this would proceed given that an interpretable solution would presumably remain a requirement. In our experiments on text-embedding-large-3, we use PCA with some success to isolate simple human-understandable distances. However, in reality, we expect that the true notions of distance used by the language model are more complex: we find additional structure in further principal components and think it is possible that a language model could encode distances in a way that is mechanistically useful, but does not correspond to any existing human understanding of the feature. Another limitation of our approach relates to the fundamental statistical difficulty of estimating manifolds in the presence of noise (Genovese et al., 2012). Here, we have opted for a simple and interpretable approach of using the K-nearest-neighbour graph to approximate the manifold, but this is prone to short-circuits causing enormous errors in the estimated manifold distances. It is often the case that one has to manually prune the graph in order to achieve reasonable manifold distance estimates, and we believe that more robust methodology for manifold estimation would be required to scale up our approach. 4.2 Implications for mechanistic interpretability research In mechanistic interpretability, one of the underlying motivations for understanding representation geometry is to be able to steer model outputs by making interventions on their internal representations. For features represented on manifolds, our insights suggest a path forward for doing this: learn the map ϕsubscriptitalic-ϕ _ fϕtypewriter_f which maps the feature subscriptZ_ fZtypewriter_f onto its representation manifold ℳsubscriptℳM_ fMtypewriter_f. Sparse autoencoders provide a potentially promising avenue for this. We conjecture that the sparsity penalty of a sparse autoencoder, trained on representation manifold, will encourage it to learn a collection of dictionary vectors which trace the manifold (see Section 4 of Engels et al. (2025) for an argument for this phenomenon on representation subspaces). We also conjecture that much observed feature splitting in SAEs is a result of this. We hope our work will encourage development of “manifold-aware” SAEs. Finally, mechanistic interpretability is a nascent field of research which is still developing a common language, and we hope that researchers will find the formalism of a feature as a metric space to be a useful possibility in future scientific discourse. References Alain and Bengio (2017) Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=HJ4-rAVtl. Ansuini et al. (2019) Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019. Arora et al. (2018) Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear Algebraic Structure of Word Senses, with Applications to Polysemy. Transactions of the Association for Computational Linguistics, 6:483–495, December 2018. ISSN 2307-387X. doi: 10.1162/tacl_a_00034. URL https://direct.mit.edu/tacl/article/43451. Bostrom (2014) Nick Bostrom. Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Oxford, UK, 2014. ISBN 978-0-19-967811-2. Bricken et al. (2023) Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread, 2023. URL https://transformer-circuits.pub/2023/monosemantic-features/index.html. Burago et al. (2001) Dmitri Burago, Yuri Burago, Sergei Ivanov, and others. A course in metric geometry. American Mathematical Society, 2001. Bussmann et al. (2024) Bart Bussmann, Michael Pearce, Patrick Leask, Joseph Bloom, Lee Sharkey, and Neel Nanda. Showing SAE Latents Are Not Atomic Using Meta-SAEs. Less Wrong, 2024. URL https://w.lesswrong.com/posts/TMAmHh4DdMr4nCSr5/showing-sae-latents-are-not-atomic-using-meta-saes. Bussmann et al. (2025) Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning Multi-Level Features with Matryoshka Sparse Autoencoders. arXiv preprint arXiv:2503.17547, 2025. Cai et al. (2021) Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International conference on learning representations, 2021. Chang et al. (2022) Tyler A. Chang, Zhuowen Tu, and Benjamin K. Bergen. The Geometry of Multilingual Language Model Representations. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 119–136. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022.EMNLP-MAIN.9. URL https://doi.org/10.18653/v1/2022.emnlp-main.9. Chatterjee (2021) Sourav Chatterjee. A new coefficient of correlation. Journal of the American Statistical Association, 116(536):2009–2022, 2021. Publisher: Taylor & Francis. Cunningham et al. (2023) Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023. Elad (2010) Michael Elad. Sparse and redundant representations: from theory to applications in signal and image processing. Springer Science & Business Media, 2010. Elhage et al. (2021) Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread, 2021. URL https://transformer-circuits.pub/2021/framework/index.html. Elhage et al. (2022) Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy Models of Superposition. Transformer Circuits Thread, 2022. URL https://transformer-circuits.pub/2022/toy_model/index.html. Engels et al. (2025) Joshua Engels, Eric J. Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not All Language Model Features Are One-Dimensionally Linear. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=d63a4AM4hb. Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6894–6910. Association for Computational Linguistics, 2021. doi: 10.18653/V1/2021.EMNLP-MAIN.552. URL https://doi.org/10.18653/v1/2021.emnlp-main.552. Genovese et al. (2012) Christopher R Genovese, Marco Perone-Pacifico, Isabella Verdinelli, and Larry Wasserman. Minimax manifold estimation. The Journal of Machine Learning Research, 13(1):1263–1291, 2012. Gorton (2024) Liv Gorton. Curve Detector Manifolds in InceptionV1, August 2024. URL https://livgorton.com/curve-detector-manifolds/. Gurnee and Tegmark (2023) Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023. Hanna et al. (2023) Michael Hanna, Roberto Zamparelli, David Mareček, and others. The functional relevance of probed information: A case study. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 835–848. Association for Computational Linguistics, 2023. He et al. (2024) Tianyu He, Darshil Doshi, Aritra Das, and Andrey Gromov. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. Advances in Neural Information Processing Systems, 37:13244–13273, 2024. Higgins et al. (2018) Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230, 2018. Hyvärinen et al. (2024) Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equation models: from linear to nonlinear. Annals of the Institute of Statistical Mathematics, 76(1):1–33, 2024. Publisher: Springer. Leask et al. (2024) Patrick Leask, Bart Bussmann, and Neel Nanda. Calendar feature geometry in GPT-2 layer 8 residual stream SAEs. Less Wrong, 2024. URL https://w.lesswrong.com/posts/WsPyunwpXYCM2iN6t/calendar-feature-geometry-in-gpt-2-layer-8-residual-stream. Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the Sentence Embeddings from Pre-trained Language Models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9119–9130. Association for Computational Linguistics, 2020. doi: 10.18653/V1/2020.EMNLP-MAIN.733. URL https://doi.org/10.18653/v1/2020.emnlp-main.733. Li et al. (2023) Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR, 2023. Publisher: ICLR. Li et al. (2025) Yuxiao Li, Eric J Michaud, David D Baek, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure. Entropy, 27(4):344, 2025. Publisher: MDPI. Lin (2024) Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2024. URL https://w.neuronpedia.org. Liu et al. (2022) Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651–34663, 2022. Mendel (2024) Jake Mendel. SAE feature geometry is outside the superposition hypothesis. AI Alignment Forum, 2024. URL https://w.alignmentforum.org/posts/MFBTjb2qf3ziWmzz6/sae-feature-geometry-is-outside-the-superposition-hypothesis. Munroe (2010) Randall Munroe. XKCD Color Name Survey Results, 2010. URL https://xkcd.com/color/rgb/. Nanda et al. (2023a) Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a. URL https://openreview.net/forum?id=9XFSbDPmdW. Nanda et al. (2023b) Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941, 2023b. Olah (2024) Chris Olah. What is a Linear Representation? What is a Multidimensional Feature? Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/july-update/index.html#linear-representations. Olah and Batson (2024) Chris Olah and Josh Batson. Feature Manifold Toy Model. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2023/may-update/index.html#feature-manifolds. Park et al. (2024) Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. arXiv preprint arXiv:2406.01506, 2024. Pfau et al. (2020) David Pfau, Irina Higgins, Alex Botev, and Sébastien Racanière. Disentangling by subspace diffusion. Advances in Neural Information Processing Systems, 33:17403–17415, 2020. Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 2019. Reizinger et al. (2025) Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E. Vogt, Randall Balestriero, Wieland Brendel, and David A. Klindt. Cross-Entropy Is All You Need To Invert the Data Generating Process. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=hrqNOxpItr. Shai et al. (2024) Adam S. Shai, Lucas Teixeira, Alexander Gietelink Oldenziel, Sarah Marzen, and Paul M. Riechers. Transformers Represent Belief State Geometry in their Residual Stream. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.c/paper_files/paper/2024/hash/8936fa1691764912d9519e1b5673ea66-Abstract-Conference.html. Smith (2024) Lewis Smith. The ‘strong’ feature hypothesis could be wrong. AI Alignment Forum, 2024. URL https://w.alignmentforum.org/posts/tojtPCCRpKLSHBdpn/the-strong-feature-hypothesis-could-be-wrong. Smolensky (1990) Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2):159–216, 1990. Publisher: Elsevier. Soares and Fallenstein (2017) Nate Soares and Benya Fallenstein. Agent foundations for aligning machine intelligence with human interests: a technical research agenda. The technological singularity: Managing the journey, pages 103–125, 2017. Publisher: Springer. Sutherland (2009) Wilson A. Sutherland. Introduction to Metric and Topological Spaces. Oxford University Press, 2009. Templeton et al. (2024) Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Christopher Olah, and Tom Henighan. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/scaling-monosemanticity/. Wang et al. (2023) Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, Anima Anandkumar, Karianne Bergen, Carla P. Gomes, Shirley Ho, Pushmeet Kohli, Joan Lasenby, Jure Leskovec, Tie\-Yan Liu, Arjun Manrai, Debora Marks, Bharath Ramsundar, Le Song, Jimeng Sun, Jian Tang, Petar Veličković, Max Welling, Linfeng Zhang, Connor W. Coley, Yoshua Bengio, and Marinka Zitnik. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47–60, August 2023. doi: 10.1038/s41586-023-06221-2. URL https://w.nature.com/articles/s41586-023-06221-2. Zhong et al. (2023) Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in neural information processing systems, 36:27223–27250, 2023. Zimmermann et al. (2021) Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International conference on machine learning, pages 12979–12990. PMLR, 2021. Appendix Appendix A Code to reproduce the experiments in this paper Code to reproduce the experiments in this paper is made available at the GitHub repository https://github.com/alexandermodell/Representation-Manifolds. Appendix B Supporting definitions and proof of Theorem 1 Hypotheses 1 and 2, and the statement of the Theorem 1 concern the metric space (,)subscriptsubscript(Z_ f, d_ f)( Ztypewriter_f , sansserif_dtypewriter_f ), and functions ϕsubscriptitalic-ϕ _ fϕtypewriter_f and gsubscriptg_ fgtypewriter_f associated with some particular feature ftypewriter_f. To de-clutter the proof we remove dependence on ftypewriter_f from the notation, and write simply (,)(Z, d)( Z , sansserif_d ), ϕitalic-ϕφϕ and g. We shall need the following definitions, informed by (Burago et al., 2001). A path in ZZ is a continuous mapping η from some interval [a,b][a,b][ a , b ] to ZZ. The length of such a path is: L⁢(η)≔sup∑i=1n⁢(ηti,ηti−1)≔subscriptsupremumsuperscriptsubscript1subscriptsubscriptsubscriptsubscript1L(η) _T _i=1^n d( _t_i, _% t_i-1)L ( η ) ≔ supcaligraphic_T ∑i = 1n sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) where the supremum is over all =(t0,t1,…,tn)subscript0subscript1…subscriptT=(t_0,t_1,…,t_n)T = ( t0 , t1 , … , titalic_n ) such that n≥11n≥ 1n ≥ 1 and t0=a≤t1≤⋯≤tn=bsubscript0subscript1⋯subscriptt_0=a≤ t_1≤·s≤ t_n=bt0 = a ≤ t1 ≤ ⋯ ≤ titalic_n = b. A path is said to be of finite length, or equivalently called rectifiable, if L⁢(η)<∞L(η)<∞L ( η ) < ∞. For [a′,b′]⊆[a,b]superscript′[a ,b ] [a,b][ a′ , b′ ] ⊆ [ a , b ] we write L⁢(η,a′,b′)superscript′L(η,a ,b )L ( η , a′ , b′ ) for the length of the restriction of η to [a′,b′]superscript′[a ,b ][ a′ , b′ ]. Any rectifiable path η:[a,b]→:→η:[a,b] η : [ a , b ] → Z admits a unit-speed parameterisation, meaning η has the representation η=η~∘φ~η= η η = over~ start_ARG η end_ARG ∘ φ where η~:[0,L⁢(η)]→:~→0 η:[0,L(η)] ~ start_ARG η end_ARG : [ 0 , L ( η ) ] → Z is a path, φ φ is a continuous, nondecreasing map from [a,b][a,b][ a , b ] to [0,L⁢(η)]0[0,L(η)][ 0 , L ( η ) ], and L⁢(η~,s,t)=t−s~L( η,s,t)=t-sL ( over~ start_ARG η end_ARG , s , t ) = t - s (Burago et al., 2001, Prop. 2.5.9). Adopting this parameterisation does not change overall length of the path, in the sense that the stated properties of η~~ ηover~ start_ARG η end_ARG imply L⁢(η~)=L⁢(η~,0,L⁢(η))=L⁢(η)~~0L( η)=L( η,0,L(η))=L(η)L ( over~ start_ARG η end_ARG ) = L ( over~ start_ARG η end_ARG , 0 , L ( η ) ) = L ( η ). We shall also consider paths on the unit hyper-sphere D−1≔x∈ℝD−1:‖x‖2=1≔superscript1conditional-setsuperscriptℝ1subscriptnorm21S^D-1 \x ^D-1:\|x\|_2=1\blackboard_SD - 1 ≔ x ∈ blackboard_RD - 1 : ∥ x ∥2 = 1 . The length of such a path, i.e., a continuous mapping γ:[a,b]→D−1:→superscript1γ:[a,b] ^D-1γ : [ a , b ] → blackboard_SD - 1 is: L⁢(γ)≔sup∑i=1n‖γti−γti−1‖2,≔subscriptsupremumsuperscriptsubscript1subscriptnormsubscriptsubscriptsubscriptsubscript12L(γ) _T _i=1^n\| _t_i- _t_% i-1\|_2,L ( γ ) ≔ supcaligraphic_T ∑i = 1n ∥ γitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT - γitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ∥2 , where again the supremum is over all =(t0,t1,…,tn)subscript0subscript1…subscriptT=(t_0,t_1,…,t_n)T = ( t0 , t1 , … , titalic_n ) such that n≥11n≥ 1n ≥ 1 and t0=a≤t1≤⋯≤tn=bsubscript0subscript1⋯subscriptt_0=a≤ t_1≤·s≤ t_n=bt0 = a ≤ t1 ≤ ⋯ ≤ titalic_n = b. Proof of Theorem 1. Since the claim of the theorem depends on η only through its length, we can assume w.l.o.g. that we are considering the unit-speed parameterisation of η. That is [a,b]=[0,L⁢(η)]0[a,b]=[0,L(η)][ a , b ] = [ 0 , L ( η ) ] and η:[0,L⁢(η)]→:→0η:[0,L(η)] η : [ 0 , L ( η ) ] → Z with L⁢(η,s,t)=t−sL(η,s,t)=t-sL ( η , s , t ) = t - s. Under Hypothesis 1, ϕitalic-ϕφϕ is continuous, hence γ:[0,L⁢(η)]→D−1:→0superscript1γ:[0,L(η)] ^D-1γ : [ 0 , L ( η ) ] → blackboard_SD - 1 defined by γt=ϕ⁢(ηt)subscriptitalic-ϕsubscript _t=φ( _t)γitalic_t = ϕ ( ηitalic_t ) is a path on D−1superscript1S^D-1blackboard_SD - 1. For any =(t0,t1,,…,tn)T=(t_0,t_1,,…,t_n)T = ( t0 , t1 , , … , titalic_n ) such that n≥11n≥ 1n ≥ 1 and t0=0≤t1≤⋯≤tn=L⁢(η)subscript00subscript1⋯subscriptt_0=0≤ t_1≤·s≤ t_n=L(η)t0 = 0 ≤ t1 ≤ ⋯ ≤ titalic_n = L ( η ), introduce the notation: S⁢(η,)≔∑i=1n⁢(ηti,ηti−1),S⁢(γ,)≔∑i=1n‖γti−γti−1‖2.formulae-sequence≔superscriptsubscript1subscriptsubscriptsubscriptsubscript1≔superscriptsubscript1subscriptnormsubscriptsubscriptsubscriptsubscript12S(η,T) _i=1^n d( _t_i, _t_i-1% ), S(γ,T) _i=1^n\| _t_i-γ% _t_i-1\|_2.S ( η , T ) ≔ ∑i = 1n sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) , S ( γ , T ) ≔ ∑i = 1n ∥ γitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT - γitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ∥2 . Fix any δ>00δ>0δ > 0. We shall prove that there exists δsubscriptT_δTitalic_δ such that : |L⁢(γ)−2⁢g′⁢(0)⁢L⁢(η)|2superscript′0 |L(γ)- -2g (0)L(η) || L ( γ ) - square-root start_ARG - 2 g′ ( 0 ) end_ARG L ( η ) | ≤|L⁢(γ)−S⁢(γ,δ)|absentsubscript ≤ |L(γ)-S(γ,T_δ) |≤ | L ( γ ) - S ( γ , Titalic_δ ) | (2) +|S⁢(γ,δ)−2⁢g′⁢(0)⁢S⁢(η,δ)|subscript2superscript′0subscript + |S(γ,T_δ)- -2g (0)% S(η,T_δ) |+ | S ( γ , Titalic_δ ) - square-root start_ARG - 2 g′ ( 0 ) end_ARG S ( η , Titalic_δ ) | (3) +−2⁢g′⁢(0)⁢|S⁢(η,δ)−L⁢(η)|2superscript′0subscript + -2g (0) |S(η,T_δ)-L(% η) |+ square-root start_ARG - 2 g′ ( 0 ) end_ARG | S ( η , Titalic_δ ) - L ( η ) | (4) ≤δ3+δ3+δ3,absent333 ≤ δ3+ δ3+ δ3,≤ divide start_ARG δ end_ARG start_ARG 3 end_ARG + divide start_ARG δ end_ARG start_ARG 3 end_ARG + divide start_ARG δ end_ARG start_ARG 3 end_ARG , (5) which implies the claim of the theorem. We shall construct δsubscriptT_δTitalic_δ in the form δ≔δ(1)∪δ(2)∪δ(3)≔subscriptsuperscriptsubscript1superscriptsubscript2superscriptsubscript3T_δ _δ^(1) _δ% ^(2) _δ^(3)Titalic_δ ≔ Titalic_δ( 1 ) ∪ Titalic_δ( 2 ) ∪ Titalic_δ( 3 ), i.e., δ(i)⊆δsuperscriptsubscriptsubscriptT_δ^(i) _δTitalic_δ( i ) ⊆ Titalic_δ for i=1,2,3123i=1,2,3i = 1 , 2 , 3, where δ(i)superscriptsubscriptT_δ^(i)Titalic_δ( i ) are defined in the remainder of the proof. We first consider a difference of the form |S⁢(γ,⋅)−2⁢g′⁢(0)⁢S⁢(η,⋅)|⋅2superscript′0⋅ |S(γ,·)- -2g (0)S(η,·) || S ( γ , ⋅ ) - square-root start_ARG - 2 g′ ( 0 ) end_ARG S ( η , ⋅ ) | as appears in (3). Noting that the mapping ϕitalic-ϕφϕ by definition satisfies ‖ϕ⁢(z)‖2=1subscriptnormitalic-ϕ21\|φ(z)\|_2=1∥ ϕ ( z ) ∥2 = 1 for all z, under Hypothesis 2, there exists ϵ>0italic-ϵ0ε>0ϵ > 0 such that if ⁢(z,z′)<ϵsuperscript′italic-ϵ d(z,z )< _d ( z , z′ ) < ϵ, then ⟨ϕ⁢(z),ϕ⁢(z′)⟩2=g⁢(⁢(z,z′)2)subscriptitalic-ϕitalic-ϕsuperscript′2superscriptsuperscript′2 φ(z),φ(z ) _2=g( d(z,z % )^2)⟨ ϕ ( z ) , ϕ ( z′ ) ⟩2 = g ( sansserif_d ( z , z′ )2 ). Let C>00C>0C > 0 be any finite constant such that supr≤ϵ|g′⁢(r)|≤Csubscriptsupremumitalic-ϵsuperscript′ _r≤ε|g (r)|≤ Csupitalic_r ≤ ϵ | g′ ′ ( r ) | ≤ C. Such a constant exists because g is C2superscript2C^2C2 by assumption. Let δ(2)=(t0(2)=0,t1(2),…,tn(2)(2)=L⁢(η))superscriptsubscript2formulae-sequencesuperscriptsubscript020superscriptsubscript12…superscriptsubscriptsuperscript22T_δ^(2)=(t_0^(2)=0,t_1^(2),…,t_n^(2)^(2)% =L(η))Titalic_δ( 2 ) = ( t0( 2 ) = 0 , t1( 2 ) , … , titalic_n( 2 )( 2 ) = L ( η ) ) be defined by: n(2)≔⌈3⁢C⁢|L⁢(η)|2δ∨L⁢(η)ϵ⌉,ti(2)≔in(2)⁢L⁢(η),i=0,…,n(2).formulae-sequence≔superscript23superscript2italic-ϵformulae-sequence≔superscriptsubscript2superscript20…superscript2n^(2) 3C|L(η)|^2δ L(η)% ε , t_i^(2) in^(2)L(η),% i=0,…,n^(2).n( 2 ) ≔ ⌈ divide start_ARG 3 C | L ( η ) |2 end_ARG start_ARG δ end_ARG ∨ divide start_ARG L ( η ) end_ARG start_ARG ϵ end_ARG ⌉ , titalic_i( 2 ) ≔ divide start_ARG i end_ARG start_ARG n( 2 ) end_ARG L ( η ) , i = 0 , … , n( 2 ) . Using the fact that η is unit-speed parameterised, it follows that, for 1≤i≤n(2)1superscript21≤ i≤ n^(2)1 ≤ i ≤ n( 2 ), L⁢(η,ti(2),ti−1(2))=ti(2)−ti−1(2)=L⁢(η)n(2)≤δ3⁢C⁢L⁢(η)∧ϵ.superscriptsubscript2superscriptsubscript12superscriptsubscript2superscriptsubscript12superscript23italic-ϵL(η,t_i^(2),t_i-1^(2))=t_i^(2)-t_i-1^(2)= L(η)n^% (2)≤ δ3CL(η) ε.L ( η , titalic_i( 2 ) , titalic_i - 1( 2 ) ) = titalic_i( 2 ) - titalic_i - 1( 2 ) = divide start_ARG L ( η ) end_ARG start_ARG n( 2 ) end_ARG ≤ divide start_ARG δ end_ARG start_ARG 3 C L ( η ) end_ARG ∧ ϵ . (6) Now consider any =(t0,t1,,…,tn)T=(t_0,t_1,,…,t_n)T = ( t0 , t1 , , … , titalic_n ) with n≥n(2)superscript2n≥ n^(2)n ≥ n( 2 ), t0=0subscript00t_0=0t0 = 0 , tn=L⁢(η)subscriptt_n=L(η)titalic_n = L ( η ) such that δ(2)⊆superscriptsubscript2T_δ^(2) _δ( 2 ) ⊆ T. Unit-speed parameterisation of η combined with δ(2)⊆superscriptsubscript2T_δ^(2) _δ( 2 ) ⊆ T implies: max1≤i≤n⁡L⁢(η,ti,ti−1)=max1≤i≤n⁡ti−ti−1≤max1≤i≤n(2)⁡ti(2)−ti−1(2)=max1≤i≤n(2)⁡L⁢(η,ti(2),ti−1(2)),subscript1subscriptsubscript1subscript1subscriptsubscript1subscript1superscript2superscriptsubscript2superscriptsubscript12subscript1superscript2superscriptsubscript2superscriptsubscript12 _1≤ i≤ nL(η,t_i,t_i-1)= _1≤ i≤ nt_i-t_i-1% ≤ _1≤ i≤ n^(2)t_i^(2)-t_i-1^(2)= _1≤ i≤ n^(% 2)L(η,t_i^(2),t_i-1^(2)),max1 ≤ i ≤ n L ( η , titalic_i , titalic_i - 1 ) = max1 ≤ i ≤ n titalic_i - titalic_i - 1 ≤ max1 ≤ i ≤ n( 2 ) titalic_i( 2 ) - titalic_i - 1( 2 ) = max1 ≤ i ≤ n( 2 ) L ( η , titalic_i( 2 ) , titalic_i - 1( 2 ) ) , and it follows from the definition of length and the triangle inequality that ⁢(ηti,ηti−1)≤L⁢(η,ti,ti−1)subscriptsubscriptsubscriptsubscript1subscriptsubscript1 d( _t_i, _t_i-1)≤ L(η,t_i,t_i-1)sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) ≤ L ( η , titalic_i , titalic_i - 1 ) for all i=1,…,n1…i=1,…,ni = 1 , … , n. Therefore using (6), we have: max1≤i≤n⁡⁢(ηti,ηti−1)≤δ3⁢C⁢L⁢(η)∧ϵ.subscript1subscriptsubscriptsubscriptsubscript13italic-ϵ _1≤ i≤ n d( _t_i, _t_i-1)≤ δ3% CL(η) ε.max1 ≤ i ≤ n sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) ≤ divide start_ARG δ end_ARG start_ARG 3 C L ( η ) end_ARG ∧ ϵ . (7) Using ‖ϕ⁢(z)‖2=1subscriptnormitalic-ϕ21\|φ(z)\|_2=1∥ ϕ ( z ) ∥2 = 1 for all z, γt=ϕ⁢(ηt)subscriptitalic-ϕsubscript _t=φ( _t)γitalic_t = ϕ ( ηitalic_t ), the upper bound by ϵitalic-ϵεϵ in (7) to enable the substitution ⟨ϕ⁢(ηti),ϕ⁢(ηti−1)⟩2=g⁢(⁢(ηti,ηti−1)2)subscriptitalic-ϕsubscriptsubscriptitalic-ϕsubscriptsubscript12superscriptsubscriptsubscriptsubscriptsubscript12 φ( _t_i),φ( _t_i-1) _2=g (% d( _t_i, _t_i-1)^2 )⟨ ϕ ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) , ϕ ( ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) ⟩2 = g ( sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT )2 ), and taking a Taylor expansion of g about zero, we have: 12⁢‖γti−γti−1‖2212superscriptsubscriptnormsubscriptsubscriptsubscriptsubscript122 12\| _t_i- _t_i-1\|_2^2divide start_ARG 1 end_ARG start_ARG 2 end_ARG ∥ γitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT - γitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ∥22 =1−⟨γti,γti−1⟩2absent1subscriptsubscriptsubscriptsubscriptsubscript12 =1- _t_i, _t_i-1 _2= 1 - ⟨ γitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , γitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ⟩2 =1−⟨ϕ⁢(ηti),ϕ⁢(ηti−1)⟩2absent1subscriptitalic-ϕsubscriptsubscriptitalic-ϕsubscriptsubscript12 =1- φ( _t_i),φ( _t_i-1) % _2= 1 - ⟨ ϕ ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT ) , ϕ ( ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) ⟩2 (8) =g⁢(0)−g⁢(d⁢(ηti,ηti−1)2)absent0superscriptsubscriptsubscriptsubscriptsubscript12 =g(0)-g (d( _t_i, _t_i-1)^2 )= g ( 0 ) - g ( d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT )2 ) =−g′⁢(0)⁢(ηti,ηti−1)2−g′⁢(ci)2⁢(ηti,ηti−1)4,absentsuperscript′0superscriptsubscriptsubscriptsubscriptsubscript12superscript′subscript2superscriptsubscriptsubscriptsubscriptsubscript14 =-g (0) d( _t_i, _t_i-1)^2- % g (c_i)2 d( _t_i, _t_i-1)^4,= - g′ ( 0 ) sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT )2 - divide start_ARG g′ ′ ( citalic_i ) end_ARG start_ARG 2 end_ARG sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT )4 , (9) where cisubscriptc_icitalic_i is some point in the interval [0,⁢(ηti,ηti−1)2]0superscriptsubscriptsubscriptsubscriptsubscript12[0, d( _t_i, _t_i-1)^2][ 0 , sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT )2 ]. Under Hypothesis 2, we have g′⁢(0)<0superscript′00g (0)<0g′ ( 0 ) < 0. Then using (9) and lemma 1 with α=‖γti−γti−1‖2subscriptnormsubscriptsubscriptsubscriptsubscript12α=\| _t_i- _t_i-1\|_2α = ∥ γitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT - γitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ∥2 and β=−2⁢g′⁢(0)⁢(ηti,ηti−1)2superscript′0subscriptsubscriptsubscriptsubscript1β= -2g (0) d( _t_i, _t_i-1)β = square-root start_ARG - 2 g′ ( 0 ) end_ARG sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ), |‖γti−γti−1‖−2⁢g′⁢(0)⁢(ηti,ηti−1)|≤|g′⁢(ci)|1/2⁢(ηti,ηti−1)2,normsubscriptsubscriptsubscriptsubscript12superscript′0subscriptsubscriptsubscriptsubscript1superscriptsuperscript′subscript12superscriptsubscriptsubscriptsubscriptsubscript12 |\| _t_i- _t_i-1\|- -2g (0) d(% _t_i, _t_i-1) |≤|g (c_i)|^1/2 % d( _t_i, _t_i-1)^2,| ∥ γitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT - γitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ∥ - square-root start_ARG - 2 g′ ( 0 ) end_ARG sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) | ≤ | g′ ′ ( citalic_i ) |1 / 2 sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT )2 , so that |S⁢(γ,)−2⁢g′⁢(0)⁢S⁢(η,)|2superscript′0 |S(γ,T)- -2g (0)S(η, % T) || S ( γ , T ) - square-root start_ARG - 2 g′ ( 0 ) end_ARG S ( η , T ) | ≤∑i=1n|‖γti−γti−1‖2−2⁢g′⁢(0)⁢(ηti,ηti−1)|absentsuperscriptsubscript1subscriptnormsubscriptsubscriptsubscriptsubscript122superscript′0subscriptsubscriptsubscriptsubscript1 ≤ _i=1^n |\| _t_i- _t_i-1\|_2-% -2g (0) d( _t_i, _t_i-1) |≤ ∑i = 1n | ∥ γitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT - γitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ∥2 - square-root start_ARG - 2 g′ ( 0 ) end_ARG sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) | ≤∑i=1n|g′⁢(ci)|1/2⁢(ηti,ηti−1)2absentsuperscriptsubscript1superscriptsuperscript′subscript12superscriptsubscriptsubscriptsubscriptsubscript12 ≤ _i=1^n|g (c_i)|^1/2 d( _% t_i, _t_i-1)^2≤ ∑i = 1n | g′ ′ ( citalic_i ) |1 / 2 sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT )2 ≤C⁢(max1≤i≤n⁡⁢(ηti,ηti−1))⁢∑i=1n⁢(ηti,ηti−1).absentsubscript1subscriptsubscriptsubscriptsubscript1superscriptsubscript1subscriptsubscriptsubscriptsubscript1 ≤ C ( _1≤ i≤ n d( _t_i, _t_% i-1) ) _i=1^n d( _t_i, _t_i-1).≤ C ( max1 ≤ i ≤ n sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) ) ∑i = 1n sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) . ≤C⁢L⁢(η)⁢max1≤i≤n⁡⁢(ηti,ηti−1)≤δ3,absentsubscript1subscriptsubscriptsubscriptsubscript13 ≤ CL(η) _1≤ i≤ n d( _t_i, _t_% i-1)≤ δ3,≤ C L ( η ) max1 ≤ i ≤ n sansserif_d ( ηitalic_t start_POSTSUBSCRIPT i end_POSTSUBSCRIPT , ηitalic_t start_POSTSUBSCRIPT i - 1 end_POSTSUBSCRIPT ) ≤ divide start_ARG δ end_ARG start_ARG 3 end_ARG , where the final inequality uses (7). In summary, we have shown that δ(2)⊆⇒|S⁢(γ,)−2⁢g′⁢(0)⁢S⁢(η,)|≤δ3.formulae-sequencesuperscriptsubscript2⇒2superscript′03T_δ^(2) |S(% γ,T)- -2g (0)S(η,T) |≤ % δ3.Titalic_δ( 2 ) ⊆ T ⇒ | S ( γ , T ) - square-root start_ARG - 2 g′ ( 0 ) end_ARG S ( η , T ) | ≤ divide start_ARG δ end_ARG start_ARG 3 end_ARG . (10) Now consider |L⁢(γ)−S⁢(γ,⋅)|⋅ |L(γ)-S(γ,·) || L ( γ ) - S ( γ , ⋅ ) | as appears in (2). By the definition of L⁢(γ)L(γ)L ( γ ), there exists δ(1)=(t0(1)=0,t1(1),…,tn(1)(1)=L⁢(η))superscriptsubscript1formulae-sequencesuperscriptsubscript010superscriptsubscript11…superscriptsubscriptsuperscript11T_δ^(1)=(t_0^(1)=0,t_1^(1),…,t_n^(1)^(1)% =L(η))Titalic_δ( 1 ) = ( t0( 1 ) = 0 , t1( 1 ) , … , titalic_n( 1 )( 1 ) = L ( η ) ) such that: L⁢(γ)−δ3≤S⁢(γ,δ(1))≤L⁢(γ).3superscriptsubscript1L(γ)- δ3≤ S(γ,T_δ^(1))≤ L(% γ).L ( γ ) - divide start_ARG δ end_ARG start_ARG 3 end_ARG ≤ S ( γ , Titalic_δ( 1 ) ) ≤ L ( γ ) . By applying the triangle inequality to the summands in S⁢(γ,δ(1))superscriptsubscript1S(γ,T_δ^(1))S ( γ , Titalic_δ( 1 ) ) we have for any TT with δ(1)⊆superscriptsubscript1T_δ^(1) _δ( 1 ) ⊆ T, S⁢(γ,δ(1))≤S⁢(γ,)≤L⁢(γ)superscriptsubscript1S(γ,T_δ^(1))≤ S(γ,T)≤ L(γ)S ( γ , Titalic_δ( 1 ) ) ≤ S ( γ , T ) ≤ L ( γ ), hence δ(1)⊆⇒|L⁢(γ)−S⁢(γ,)|≤δ3.formulae-sequencesuperscriptsubscript1⇒3T_δ^(1) |L(% γ)-S(γ,T) |≤ δ3.Titalic_δ( 1 ) ⊆ T ⇒ | L ( γ ) - S ( γ , T ) | ≤ divide start_ARG δ end_ARG start_ARG 3 end_ARG . (11) Now consider −2⁢g′⁢(0)⁢|S⁢(η,⋅)−L⁢(η)|2superscript′0⋅ -2g (0) |S(η,·)-L(η) |square-root start_ARG - 2 g′ ( 0 ) end_ARG | S ( η , ⋅ ) - L ( η ) | as appears in (4). By similar arguments to those used above do establish (11), there exists δ(3)superscriptsubscript3T_δ^(3)Titalic_δ( 3 ) such that δ(3)⊆⇒−2⁢g′⁢(0)⁢|L⁢(η)−S⁢(η,)|≤δ3.formulae-sequencesuperscriptsubscript3⇒2superscript′03T_δ^(3) -2g^% (0) |L(η)-S(η,T) |≤ δ3.Titalic_δ( 3 ) ⊆ T ⇒ square-root start_ARG - 2 g′ ( 0 ) end_ARG | L ( η ) - S ( η , T ) | ≤ divide start_ARG δ end_ARG start_ARG 3 end_ARG . (12) With δ≔δ(1)∪δ(2)∪δ(3)≔subscriptsuperscriptsubscript1superscriptsubscript2superscriptsubscript3T_δ _δ^(1) _δ% ^(2) _δ^(3)Titalic_δ ≔ Titalic_δ( 1 ) ∪ Titalic_δ( 2 ) ∪ Titalic_δ( 3 ), the implications (10), (11), (12) together tell us that the inequality (5) holds, and this completes the proof of the theorem. ∎ Lemma 1. For any α,β≥00α,β≥ 0α , β ≥ 0, |α−β|≤|α2−β2|1/2superscriptsuperscript2superscript212|α-β|≤|α^2-β^2|^1/2| α - β | ≤ | α2 - β2 |1 / 2. Proof. W.l.o.g., assume α≥βα≥βα ≥ β. Using the triangle inequality for the Euclidean norm in ℝ2superscriptℝ2R^2blackboard_R2, α=(β2+α2−β2)1/2≤β+(α2−β2)1/2superscriptsuperscript2superscript2superscript212superscriptsuperscript2superscript212α=(β^2+α^2-β^2)^1/2≤β+(α^2-β^2)^% 1/2α = ( β2 + α2 - β2 )1 / 2 ≤ β + ( α2 - β2 )1 / 2, i.e., α−β≤(α2−β2)1/2superscriptsuperscript2superscript212α-β≤(α^2-β^2)^1/2α - β ≤ ( α2 - β2 )1 / 2 . ∎

Instant research discovery