Paper deep dive
When Models Don't Manipulate Manifolds: The Geometry of a Comparison Task
Sai Sumedh R. Hindupur, Hadas Orgad, Thomas Fel, Demba Ba
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:One of the current premises of mechanistic interpretability research is that detailed accounts of the geometry of neural network representations can tell us how models perform computations, and how to effectively intervene on them. While low dimensional manifolds have been observed for multiple concepts in the literature (e.g. numbers encoded on helices, days of the week on a circle, ...), with structure believed to reflect properties of data and tasks, the extent to which models rely on them for computation, and how they manipulate them, remains unclear. We characterize precisely the geometry of computation in a number-comparison task, as an abstraction of comparison for decision making, and how models utilize geometry in an elegant fashion to implement it. Specifically, we study the causal geometry of number comparison in Qwen2.5-7B-Instruct, a capable and widely studied open-weight model, and find Qwen largely uses linear representations of numbers despite the presence of curved geometry. To compare two numbers, the model first encodes each number along a vector and adds the two representations using attention and the residual connection, bringing them into a shared space in the residual stream. Then, the model uses MLP neurons to compare the pair of numbers on local regions in this shared space, which correspond to smaller intervals of input numbers, and combines these to obtain the position of the maximum. In fact, this reliance on linear representations for comparison also persists when the model compares three numbers. Our findings demonstrate that the manifold hypothesis can co-exist with linear representations: while concepts that are ordered may have manifold structure in representations, the model may use an underlying linear structure of the concept in certain computations.
Tags
Links
- Source: https://arxiv.org/abs/2609.37680v1
- Canonical: https://arxiv.org/abs/2609.37680v1
Trouble viewing inline? Open PDF directly →
Full Text
86,279 characters extracted from source content.
Expand or collapse full text
WHEN MODELS DON’T MANIPULATE MANIFOLDS: THE GEOMETRY OF A COMPARISON TASK Sai Sumedh R. Hindupur 1 , Hadas Orgad 2 , Thomas Fel 3 and Demba Ba 1,2 School of Engineering and Applied Science, Harvard University 1 , Kempner Institute, Harvard University 2 , Goodfire AI 3 ABSTRACT One of the current premises of mechanistic interpretability research is that detailed accounts of the geometry of neural network representations can tell us how models perform computations, and how to effectively intervene on them. While low dimensional manifolds have been observed for multiple concepts in the literature (e.g. numbers encoded on helices, days of the week on a circle, ...), with structure believed to reflect properties of data and tasks, the extent to which models rely on them for computation, and how they manipulate them, remains unclear. We characterize precisely the geometry of computation in a number-comparison task, as an abstraction of comparison for decision making, and how models utilize geometry in an elegant fashion to implement it. Specifically, we study the causal geometry of number comparison in Qwen2.5-7B-Instruct, a capable and widely studied open-weight model, and find Qwen largely uses linear representations of numbers despite the presence of curved geometry. To compare two numbers, the model first encodes each number along a vector and adds the two representations using attention and the residual connection, bringing them into a shared space in the residual stream. Then, the model uses MLP neurons to compare the pair of numbers on local regions in this shared space, which correspond to smaller intervals of input numbers, and combines these to obtain the position of the maximum. In fact, this reliance on linear representations for comparison also persists when the model compares three numbers. Our findings demonstrate that the manifold hypothesis can co-exist with linear representations: while concepts that are ordered may have manifold structure in representations, the model may use an underlying linear structure of the concept in certain computations. 1 1INTRODUCTION How do language models represent concepts internally, and how do they use these representations to perform computations? The linear representation hypothesis (Park et al., 2023; Elhage et al., 2021) posited that models represent ordered concepts as one-dimensional subspaces, which found evidence across diverse applications (Zhu et al., 2024; Marks & Tegmark, 2024; Voynov & Babenko, 2020; Tigges et al., 2023; Lee et al., 2024). Recent work has challenged this perspective and found that these concepts live on low dimensional manifolds – a phenomenon known as the manifold hypothesis: examples include number helices (Kantamneni & Tegmark, 2025), curved manifolds for dates/years (Modell et al., 2025), character count manifolds (Gurnee et al., 2026), as well as age and temperature manifolds (Bhalla et al., 2026). Where does this manifold structure come from? Theoretical analyses suggest that the manifold structure arises from symmetries in data (Karkada et al., 2026) or task symmetries (Hwang & Park, 2026). These insights have also revealed how steering along manifolds can control model behavior (Wurgaft et al., 2026). However, given a specific computation, which aspects of the representation structure the model uses to perform the computation remains an open question. Does the model manipulate nonlinear concept manifolds, as Gurnee et al. (2026) show using character count manifolds on a line-breaking task, or does the model use simpler linear representations for certain tasks despite the presence of manifold structure? 1 Our code is available at https://github.com/Sai-Sumedh/comparison-geometry 1 arXiv:2609.37680v1 [cs.LG] 29 Sep 2026 Compare via MLPs The maximum of 3 and 7 is 3 7 InputOutput Add via attention 7 Read numbers along linesBring numbers together 3 7 First bigger Second bigger Figure 1: Linear number representations are used by language model Qwen for comparison. When asked to compare numbers, the model first represents them on a nonlinear manifold, but uses structure along a single linear direction in computation. It combines information about the two numbers by adding the corresponding linear representations using a single attention head and a residual connection, creating a shared space which enables comparison. On this shared space, the model identifies which number is larger using MLP neurons across two layers, and uses the answer position to read out the answer to the max task. Elucidating how models manipulate internal representations to perform computations has become an active area of research. Framed as such, this question amounts to investigating the algorithmic level in Marr’s levels of analysis (Marr, 2010), which claim that a system can be understood at the computational (abstract, behavioral), algorithmic (variables and how they are manipulated) and imple- mentation levels (low-level implementation of the algorithm). Mechanistic interpretability research, despite having largely focused on finding circuits of computation, namely the implementation level, has also become interested in the algorithmic level (Geiger et al., 2021) which operates at a higher level of abstraction. Multiple examples of algorithmic understanding of models exist, including trigonometry on circular/ helical structures for addition (Kantamneni & Tegmark, 2025), an addition mechanism using Fourier features for multiple concepts such as week days, months, etc (Feucht et al., 2026), and explaining attention heads with python programs (Hayes et al., 2026). Our work contributes to this burgeoning and exciting line of work on algorithmic interpretability. Motivated by the above lines of inquiry, we ask how language models implement comparison using representation geometry. Comparison is a widely useful operation for intelligent systems, and language models in particular. It is an integral component of weighing options and making decisions, especially using numerical or ordered concepts (which can be represented on a line; see (G ̈ ardenfors, 2000)). For example, queries like ”Which of these apartments is closest to my office?”, “Which quarter had the worst sales in this five-year report?” require performing comparison of the specified objects using the specified attributes. Despite reports of nonlinear, helical representations of numbers (Kantamneni & Tegmark, 2025), the aspects of number representations models use for the purpose of comparison, and how do they manipulate them to implement the operation remains poorly understood. Previous works have studied how models perform comparison. (Hanna et al., 2023) characterize in great detail a comparison circuit in GPT-2. However, they do not establish causal role of representation geometry in comparison by acknowledging that ”GPT2’s structured number representations may be relevant to its greater-than ability. However, our experiments struggle to prove this causally” (quoted from Hanna et al. (2023)). (El-Shangiti et al., 2025) found a linear subspace which causally affects model outputs, but their analysis does not concern the mechanism by which the model compares values represented in this subspace. (Yuchi et al., 2026) study mixed notation number comparison and compare behavioral accuracy with classifier performance. Taken together, these works have focused on circuit discovery, model performance, or probing accuracy: they however do not study causal representation geometry and how it is used by the model to perform comparison. We provide a detailed geometric account of the algorithm language models employ to compare numbers (Fig. 1). In addition to being descriptive, our account is causal: we can predict how causal interventions will affect model behavior. Concretely, we make the following contributions: •Reconciling linear representations and manifolds: We demonstrate how linear representations of numbers are causally involved in the model’s number comparison, despite the presence of underlying curved manifold structure. • Algorithm for number comparison using linear representations: We further show how the model manipulates number representations to perform comparison: by additive mixing of 2 Algorithm 1 Pairwise Number Comparisons Require: numbers y 1 ,y 2 at times t = 1, 2 resp. if t = 1 then store y 1 asx 1 l = uf ′ 1 (y 1 )▷ Fig. 2 (a, b) else if t = 2 then store y 2 asx 2 l =v 2 f 2 (y 2 )▷ Fig. 2 (c) copyx 1 l , transform asv 1 f 1 (y 1 ),▷ Done by Attention Head(s), Fig. 2(b, d) add to residualx 2 l+1 =x 2 l +v 1 f 1 (y 1 )▷ Residual Stream, Fig. 3 (a–c) divide span(v 1 ,v 2 ) into regionsR i ▷ MLP neurons, Fig. 4(a, b) compare y 1 ,y 2 inR i as C i = I(y 1 > y 2 ).I(x 2 l+1 ∈R i ) ▷ MLP neuron outputs, Fig. 4(b) combine local comparisonsC i to get ans = I(y 1 > y 2 ) ▷ later MLP neurons, Fig. 4(c, d) end if individual number directions, followed by local comparisons (for specific intervals of input numbers) which are then combined to give the global comparison answer. •Extension of the comparison algorithm to longer sequences of numbers: We show how the algorithm using linear number representations extends to three-length sequences. 2PAIRWISE COMPARISON OF NUMBERS In this section, we first describe the pairwise comparison mechanism in an LLM. We state the algo- rithm explicitly, and discuss the main steps involved. Furthermore, we provide evidence describing how the model implements this algorithm in subsequent subsections. We extend the algorithm to three number comparisons and include evidence in Section 3. Notation.Computationally relevant subspaces within model activations are denoted byx t l , wherel denotes the layer index within the model andtdenotes time (token position). Numbers present in the input prompt are denoted byy t . v i are directions in model activations, which belong to the same space (same layer and token position). These directions encode numbers, withv i encodingy i asv i f i (y i ), where f i may be a nonlinear function of y i . 2.1ALGORITHM FOR PAIRWISE COMPARISONS In line with the known distinction between availability and utility of features (Garg et al., 2026), our claims in the following sections are about linear features for numbers being used by the model for a specific task: comparison. Other tasks and concepts, like addition/ periodic concepts (Feucht et al., 2026; Wurgaft et al., 2026)) may use more intricate manifold structure. The model encodes each number’s magnitudey i as a nonlinear functionf i (y i )along a single direction v i , which is different for each number position. Since the two numbersy 1 ,y 2 are provided as inputs at different times (distinct token positions), the model then creates a shared representation from the two numbers by (1) copying information abouty 1 intoy 2 ’s position, and (2) adding together the single number representations. The shared representation is expressed as: x =v 1 f 1 (y 1 ) +v 2 f 2 (y 2 )(1) This representation spans a two-dimensional planespan(v 1 ,v 2 ). Moving along certain directions in this plane (e.g., alongαv 1 − βv 2 for anyα > 0,β ≥ 0) changes the probability of the model answeringy 1 as the greater number. The comparisony 1 > y 2 can be linearly decoded on this plane. However, the model implements comparison in two stages, as described below. From this shared representation, MLP neurons first perform local comparisons, identifying and comparing the two numbersy 1 ,y 2 for specific ranges of individual numbers or their combinations. This occurs because of the gating-based nonlinearity of MLP neurons (SwiGLU for Qwen2.5-7B), whose sigmoid gate and overall expression leads to local regions of activation on intersection with 3 2010010 PC1 (16%) 20 10 0 10 PC3 (7%) (a) 20 40 60 80 y 1 20406080100 y 1 20 15 10 5 0 5 10 f ′ 1 ( y 1 ), comp. along u (b) along u along v 1 4 2 0 2 f 1 ( y 1 ), comp. along v 1 20406080100 y 2 30 20 10 0 10 f 2 ( y 2 ), comp. along v 2 (c) y 1 position token position 0.0 0.2 0.4 0.6 H14 attention (d) 255075100 y 1 0.2 0.4 0.6 0.8 attention to y 1 H14H18H20H13H11 layer 14 heads (top 5 of 28) 0.0 0.1 0.2 0.3 0.4 PR (pos. recovery) (e) v 2 alone head alone head & v 2 y 1 y 2 number perturbed 0.00 0.25 0.50 0.75 1.00 1.25 IIA full 0.07 u 0.94 v 1 0.46 full 0.98 v 2 0.79 (f) Figure 2: A single causal direction for numbers controls model comparison. (a) A causal direction ufound in Layer 13’s residual stream causally affects model behavior, despite the presence of curved geometry in the activations, as observed in principal components 1, 3. (b, c) Projection of activations onto the obtained causal directionsu,v 1 ,v 2 encode the number magnitude for a wide range of values. Whileuis in layer 13 residual stream at the first numbery 1 ’s position,v 1 ,v 2 are causal directions encodingy 1 ,y 2 resp. aty 2 ’s position.v 1 is obtained using DAS at an attention head H14’s outputs. (d) Attention head 14 in layer 14 attends to the position of the first numbery 1 , irrespective of the number value, serving as a copy head. (e) Among all the attention heads in Layer 14, head H14 is causally involved in the model’s computation, showing significantly higher position recovery than others. (f) Patching alongu,v 1 ,v 2 has significant causal effects on model behavior, as shown by interchange intervention accuracy (IIA). span(v 1 ,v 2 ). The outputs of the neurons are nonlinear on local regions, a consequence of approximate quadratic behavior of the SwiGLU nonlinearity on active regions. Subsequent MLP neurons then combine these local comparisons to create global comparator neurons, which nearly perfectly captureI(y 1 > y 2 ). These neurons then construct a single direction in the residual stream which encodes the comparison’s answer, and causally affects the model’s outputs. The algorithm is stated in Alg. 1, visualized in App. Fig. 9, along with evidence demonstrating each step. Evidence for the pairwise comparison algorithm. We perform experiments using the open-weight model Qwen2.5-7B-Instruct (Yang et al., 2024). We ask the model to compare pairs of numbers, and provide the model with a one-shot example for output format. The prompt is ”Answer in the following format with a single answer. The maximum of 12 and 4 is 12. The maximum of y1 and y2 is ”. Our causal analyses involve activation patching using interchange interventions, where we patch specific component activations from a model running on a ’clean’ prompt to when the model is processing another ’corrupt’ prompt ((Meng et al., 2022)). For example, suppose the ’clean’ prompt has inputs(60, 10). The corrupt prompt then uses(3, 10)as inputs, and patching activations from clean to corrupt changes the model’s outputs on the ’corrupt’ prompt. In this example, since the first number is changed between the clean and corrupt prompts, we refer to this as ’y 1 perturbed’ in subsequent figures Fig. 2, 3, 4, 5. Further insights into our experimental setup is included in App. A. The model computes the argmax position and uses that to produce the answer. First, we observe that when patching is successful, patching model activations from a ’clean’ run to a ’corrupted’ run leads to changing the model’s answer position, instead of the value (see App. C.1). For instance, patching activations from(60, 10)into a model processing(3, 10)will make the model answer3, and not60. Therefore, internal model activations compute thearg maxposition and use that to produce the answer. We restrict our analysis to the model’s computation of the arg max position. 4 505 f 1 (y 1 ), comp. along v 1 20 10 0 10 20 comp. along v 2 (a) v 1 v 2 = 94.7 20 40 60 80 y 1 505 f 1 (y 1 ), comp. along v 1 20 10 0 10 20 comp. along v 2 (b) v 1 v 2 20 40 60 80 y 2 505 f 1 (y 1 ), comp. along v 1 20 10 0 10 20 comp. along v 2 (c) v 1 v 2 no yes y 1 > y 2 u,v 1 u,v 2 v 1 ,v 2 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 cosine similarity (d) random pairs (±2 ) z 2 l = 13 v 2 v 1 v 1 &v 2 v 1 ,v 2 plane z 2 l = 14 patched component 0.0 0.2 0.4 0.6 0.8 1.0 1.2 IIA (e) y 1 perturbedy 2 perturbed 505 f 1 (y 1 ), comp. along v 1 20 10 0 10 20 comp. along v 2 (f) v 1 v 2 0.10 0.25 0.75 0.90 0.50 0.00 0.25 0.50 0.75 1.00 P(model answer = y 1 ) Figure 3: The model uses causal number directionsv 1 ,v 2 to construct a shared two-dimensional representation encoding the two inputsy 1 ,y 2 . (a)–(c) The linear span ofv 1 andv 2 shows each number is encoded along its own direction, and the answer to comparison is linearly separable in this shared representation space. (d) By copyingy 1 fromutov 1 , the model reduces the alignment between the two number representations:| cos(v 1 ,v 2 )| << | cos(u,v 2 )|. (e) While the residual stream at layer 13, denotedz 2 l=13 , encodesy 2 ,v 1 brings in causally useful information into they 2 position. The shared representation is additive: Patchingv 1 &v 2 together (two rank-one patches) nearly matches the IIA of the rank-two patch onto thev 1 ,v 2 plane (i.e.,span(v 1 ,v 2 )). The plane itself captures as much information as the next layer 14 residual stream. (f) Moving around thev 1 ,v 2 plane, which is a two-dimensional plane in the 3,584-dimensional residual stream, is sufficient to change model behavior predictably. We demonstrate the pairwise comparison algorithm by analyzing individual number representations (Fig. 2), which combine to form the shared representation (Fig. 3). MLP neurons then perform the comparison in two stages, first locally and then globally (Fig. 4). 2.2INDIVIDUAL NUMBER REPRESENTATIONS Setup. Using 2000 ordered pairs of two-digit numbers, which include 1000 unique randomly chosen pairs(a,b)and their reflections(b,a), we collect model activations at all layers and all token positions while processing the input prompt. The pairsa,bare chosen to have distinct leading digits, so that the effect of patching is visible at the model logits since tokens are individual digits (two digit numbers having different first digits are a large fraction of all possible pairs,∼ 90%). We use both PCA and Distributed Alignment Search (DAS) (Geiger et al., 2024) to find causally relevant directions encoding each numbery 1 ,y 2 : we employ DAS whenever the principal components are not causally relevant. Observations. Fig. 2a shows that despite nonlinear manifold structure of number representations (shown in PC1-PC3 projection), there exists a directionuin layer 13’s residual stream which causally affects the model’s answer, as measured by Interchange Intervention Accuracy (IIA, (Geiger et al., 2021)) (Fig. 2f). Note that we compute IIA using the immediate next token generated by the model. However, the IIA scores are very similar for patching along directions and subspaces of interest even when computed using the entire number generated by the model (see App. C.7). A single attention head H14 of layer 14 copies information abouty 1 from layer 13 in token positiony 1 toy 2 . It consistently attends to they 1 position (Fig. 2 d), and patching this head’s output has the highest effect on the model’s logits, as measured using position recovery (a modified version of recovery 5 505 f 1 (y 1 ), comp. along v 1 30 20 10 0 10 20 comp. along v 2 (a) v 1 v 2 20 50 80 y 2 #6262 (b) #6150#5076 205080 y 1 20 50 80 y 2 #9459 205080 y 1 #6854 205080 y 1 #15775 205080 y 1 20 50 80 y 2 #12784 (c) 1050510 rank-1 DAS component, L15 residual 0 20 40 60 80 100 count (d) y 1 > y 2 y 2 > y 1 (v 1 ,v 2 ) patch (v 1 ,v 2 ) patch + freeze top-12 MLP neurons L15 DAS rank 1 0.0 0.2 0.4 0.6 0.8 1.0 1.2 IIA (e) y 1 perturbedy 2 perturbed 1 0 1 activation / peak 1 0 1 Figure 4: The model compares numbersy 1 ,y 2 by combining local comparisons on the shared v 1 ,v 2 plane. (a) Individual neuron weights in layer 14 MLP are specific directions in thev 1 ,v 2 plane. The y-axis isv ⊥ 2 , the component ofv 2 orthogonal tov 1 (sincev 1 ,v 2 are not exactly orthogonal) (b) Neurons in layer 14 MLP (ranked by attribution scores) localize specific regions of the inputs y 1 ,y 2 (like neuron#6150), or perform comparisons in localized regions (neuron#9459). (c) Select neurons in layer 15 MLP, which combine the outputs of layer 14 MLP neurons, are global comparators: they respond positively wheny 2 > y 1 and negatively otherwise. (d) A single direction in the layer 15 residual stream (after layer 15 MLP), which is formed by inputs from global comparator neurons from layer 15 MLP, encodes the position of the answerarg max(y 1 ,y 2 ). (e) There are 12 comparator neurons (6 each in layer 14, 15 MLPs) which perform comparison: freezing these neurons significantly degrades the IIA achieved by patching in the(v 1 ,v 2 )plane . The one-dimensional comparison direction in layer 15 (panel (d)) controls model behavior. (Meng et al., 2022) that patching changes the model’s answer position instead of value). PR = LD patch − LD corrupt LD clean − LD corrupt (2) where LD is the logit difference between the pair(min(y 1 ,y 2 ), max(y 1 ,y 2 ))wherey 1 ,y 2 are inputs on the corrupt prompt, since upon patching from clean to corrupt prompts, models outputmin(y 1 ,y 2 ) as the answer to the corrupt max-prompt. Note that we call this position recovery to observe how well the model reorganizes its logits to patching and answers with the patched position. The denominator is only meant to provide a rough scale of logit difference.v 1 is then obtained as the DAS direction at the output of H14.v 2 is obtained as the top principal component from the layer 13 residual stream. Therefore, obtained causal directionsu,v 1 ,v 2 encode the values ofy 1 ,y 1 ,y 2 respectively, as shown in panels (b, c). These directions are causal, as shown by their IIA scores in (Fig. 2f). 2.3SHARED REPRESENTATION ENABLES COMPARISON Using the causal directions of individual numbersv 1 ,v 2 , the model creates a shared representation of both numbers, which is a two-dimensional plane in the pre-MLP residual stream of layer 14. We visualize the projection of model activations onto the linear span of these directions (Fig. 3). This shared representation lives in the pre-MLP residual stream of layer 14. Fig. 3 (a)–(c) show activations in this space encoding each number along its own direction, while allowing linear separation of the two casesy 1 > y 2 ,y 1 < y 2 . The probability contours of the model’s answer are approximately parallel to the decision boundary (Fig. 3 f). The directionsv 1 ,v 2 are nearly orthogonal (Fig. 3 d) and 6 Algorithm 2 Three-Number Comparisons Require: numbers y 1 ,y 2 ,y 3 at times t = 1, 2, 3 resp., ̃ T is the last token. if t = 1 then store y 1 asx 1 l = u 1 f 1 (y 1 ) else if t = 2 then store y 2 asx 2 l = u 2 f 2 (y 2 ) computeα 2 = I(y 2 = max(y 1 ,y 2 ))and store alongd 2 ▷Use Alg. 1; Fig. 5 (d), App. Fig. 26 else if t = 3 then store y 3 asx l (3) =v 3 f 3 (y 3 )▷ Fig. 5(a) copyx 1 l ,x 2 l , transform asv 1 f 1 (y 1 ),v 2 f 2 (y 2 ) ▷ Done by Attention Heads, App. Fig. 26 add to residualx l+1 (3) =x l (3) +v 1 f 1 (y 1 ) +v 2 f 2 (y 2 ) ▷ Residual Stream, App. Fig. 26 divide span(v 1 ,v 2 ,v 3 ) into regionsR i ▷ Fig. 5(c) compare inR i as C i = I(y 3 = max(y 1 ,y 2 ,y 3 )).I(x l+1 (3)∈R i ) ▷ MLP neurons, Fig. 5(c) combineC i , getα 3 = I(y 3 = max(y 1 ,y 2 ,y 3 )), store ond 3 ▷Later MLP neurons, Fig. 5(d) else if t = ̃ T then combine α 2 d 2 ,α 3 d 3 to getx ℓ ′ ( ̃ T) = P t w t α t d t ▷ Attention Head, Fig. 5(e), App. Fig. 26 Read arg max(y 1 ,y 2 ,y 3 ) fromx ℓ ′ ( ̃ T) end if less aligned than their counterparts which lived at different token positions (u,v 2 ). Patching these directions using two rank-one patches (i.e., by projecting alongv 1 andv 2 ) shows a similar causal effect (IIA) as patching the rank-two shared representation space (by projecting ontospan(v 1 ,v 2 )) (Fig. 3 e), indicating linear combination of individual number representations create this shared space. This observation is nontrivial becausev 2 is obtained in layer 13’s residual stream andv 1 is from head H14’s output (which itself uses the layer 13 residual stream in computation), making nonlinear interactions possible. 2.4GLOBAL COMPARISON IS A COMBINATION OF LOCAL COMPARISONS Setup. We first identify which layer MLPs are involved in the maximum computation by performing freezing experiments. Here, we patch thespan(v 1 ,v 2 )space, but freeze the downstream MLP outputs to remain the same as the no-patch case. This allows us to check the contribution of individual layer MLPs in the subsequent computation: we expect that freezing important MLPs will result in a significant drop in the patching effectiveness. For relevant MLPs, we then identify neurons of interest using first-order attribution scores from the model logits (App. B.5, App. C.5). The obtained neurons are tested for causal relevance by further patch-and-freeze experiments (see App. B for details). Observations. MLP neurons in layers 14 and 15 operate on the shared representation space span(v 1 ,v 2 )(Fig. 4(a)). We find a set of 12 neurons (6 neurons in MLP of layer 14 and 6 in layer 15’s MLP) which together contribute to the comparison computation, and freezing these neu- rons significantly reduces the IIA from patching the(v 1 ,v 2 )space (see Fig. 4 (e) and App. C.5). They perform the comparison in two stages: first, neurons in the MLP of layer 14 perform localized comparisons: they respond to specific ranges of values ofy 1 ory 2 , or a combination thereof (e.g., |y 1 − y 2 | < η) (see Fig. 4 (b) which shows the receptive fields of neurons, i.e., their activations as a heatmap in the(y 1 ,y 2 )space). While the geometry of thespan(v 1 ,v 2 )space makes comparison possible by linear readouts (Fig. 3c), MLP neurons seem to use their linear transform followed by nonlinearity to perform local comparisons. Some neurons perform the comparisony 1 < y 2 on these bounded ranges (like neuron#9459in Fig.4(b)) while others serve to identify the ranges (like neuron#6150in Fig. 4(b)). These neurons are then combined in the second stage (MLP of layer 15), into neurons which serve as global comparators, encoding the answer position over the entire range of values of both numbers (Fig. 4(c)). We find a single direction (using DAS) in layer 15’s residual stream (after MLP of layer 15), which is written to by several global comparator neurons, and encodes the answer to the comparison as a binary value (Fig.4(d)). The importance of these twelve MLP neurons, as well as the causal relevance of the DAS direction obtained in layer 15, are shown using IIA in panel (Fig. 4(e)). 7 20406080 y j 1.5 1.0 0.5 0.0 0.5 1.0 comp. along v j (z-score) (a) v 1 vs y 1 v 2 vs y 2 v 3 vs y 3 v 1 v 2 v 3 span (v 1 ,v 2 ,v 3 ) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 PR (b) y 1 perturbed y 2 perturbed y 3 perturbed y 1 y 2 (c) y 1 y 3 MLP14 #5076 y 2 y 3 y 1 y 2 y 1 y 3 MLP15 #12784 y 2 y 3 0 50 100 count at y 2 (d) y 2 max y 2 not max 1050510 flag 0 50 100 count at y 3 y 3 max y 3 not max 201001020 PC1 (L20, last token) 20 10 0 10 PC2 (e) max y 1 max y 2 max y 3 flag at y 2 flag at y 3 answer y 1 > y 2 answer y 1 > y 3 answer y 2 > y 3 0 1 2 3 PR (f) L20 PC1,2 full Figure 5: The model continues to use linear number representations for three-number compar- isons. (a) We observe that the causal directionsv 1 ,v 2 ,v 3 (obtained at the third numbery 3 ’s position) encode the numbersy 1 ,y 2 ,y 3 , through nearly monotonic componentsf i (y i ). (b) The three directions v 1 ,v 2 ,v 3 have causal effects on the model outputs (logits, measured using position recovery, PR). The effect ony 2 is lower at this position (y 3 position). (c) Receptive fields from layer 14, layer 15 MLP neurons show local comparisons now being performed in they 1 ,y 2 ,y 3 space (whose three two-dimensional views are shown). (d) The model represents its answer flag along a single direction in the layer 15 residual stream at both they 2 and they 3 position. The flag identifies ify t is the largest upto timet. Note thaty 2 maxrefers toy 2 > y 1 here. (e) Thearg maxposition is represented in the top two principal components of layer 20 residual stream, at the last token position (after the third number y 3 , and right before the model answers). (f). The flags at y 2 and y 3 positions, as well as the answer position from (e), are causal and affect model logits. 3MULTIPLE COMPARISONS: THE CASE WITH THREE NUMBERS Having stated and established evidence for the pairwise comparisons algorithm in Qwen, we now extend it to multiple number comparisons: does the model continue to use linear representations of numbers for multiple comparisons? The model is very good at the multiple comparison task: comparing long sequences of numbers in a single forward pass (see App. C.1). Using three numbers as a case study, we state the model’s algorithm and describe evidence for this algorithm in model activations. 3.1ALGORITHM FOR MULTIPLE COMPARISONS To compare longer sequences of numbers, language models employ an interesting strategy: at every timet, they compute a binary flag which answers the following question: isy t the largest number seen so far? To achieve this, they construct an additive sum of linear representations of all numbers up to and includingy t . Individual neurons then perform local comparisons on this shared representation, which is combined to give a global comparator that identifies whethery t is the largest entry seen so far. This information is stored along a single directiond t , as a binary variable α t = I(y t = max(y 1 ,...,y t )). x =v 1 f 1 (y 1 ) +v 2 f 2 (y 2 ) +v 3 f 3 (y 3 ), at t, α t = I(y t = max(y 1 ,...,y t )). (3) At this stage of computation, ify t is not the maximum upto timet, i.e., the flagα t = 0, the position of the maximum is not otherwise stored (i.e., the model simply knows that the maximum is not 8 att, but not if the maximum is at an intermediate position). Information about whether the first numbery 1 (which is a special case since there are no numbers before it) is the maximum is also computed at they 3 position. At the final token ̃ T before generating the answer, the model combines these directions using a single attention head which attends to the answer position (i.e.,w t is high at arg max(y 1 ,...,y T )) to obtainx ℓ ′ ( ̃ T) = P T t=1 w t α t d t (App. Fig. 26). This final representation encodes the answer position which is then read off by the model. While our results describe how individual number representations are used to create causally relevant flags, we empirically find the above attention head. We leave an investigation into the complete mechanism of how the head works, and how the model decodes the answer position to future work. We hypothesize that the model may read this information by exploiting the order of numbers, checking sequentially (in decreasingt) ifα t = 1and identifying the first time this occurs. We state the algorithm in Alg. 2. 3.2EVIDENCE FOR THE THREE NUMBER COMPARISON ALGORITHM Setup. We extend our analysis to studying comparison of three numbers. We choose1500triples and400held-out examples. In this case, IIA does not show enough signal for analysis, so we employ position recovery (Eq. 2) to test for causal effects of activation patching on the model’s logits. Observations. First, we note that when given three numbers, the model seems to perform two parallel computations (see App. C.6). Fig. 5’s first row shows the computation occurring at the third number position. Using three directionsv 1 ,v 2 ,v 3 which encode the magnitudes of the three numbersy 1 ,y 2 ,y 3 (panel a), the model creates a three-dimensional spacespan(v 1 ,v 2 ,v 3 ). Fig. 5(b) shows that patching each direction affects the model outputs when the corresponding number is perturbed, and that patching the span affects multiple numbers. The effect of perturbingy 2 is minimal here, indicating that the model uses different computation for determiningy 2 is the largest. Representative MLP neurons in layer 14 MLP perform local comparisons in the(y 1 ,y 2 ,y 3 )space (Fig. 5(c)). These neurons construct a single direction in layer 15’s residual stream space which encodes the answerI(y 3 = max(y 1 ,y 2 ,y 3 ))(Fig. 5(d), bottom). At the second numbery 2 ’s position, the model comparesy 2 andy 1 (using Alg. 1), and creates a single direction from DAS which encodes the answer position in layer 15’s residual stream (Fig. 5(d), top). At the last token position, layer 20’s residual stream combines the flag information (using an attention head, see App. Fig. 26) encodes the position of the answer, as visible on a two dimensional PCA Fig. 5(e). Position recovery scores in Fig. 5(f) show that the flags at they 2 andy 3 positions, as well as the two-dimensional subspace encoding the overall answer in layer 20 (from Fig. 5(e)) are causal. The two principal components in layer 20 have the same position recovery as the entire residual stream at that position (Fig. 5(f)). 4DISCUSSION AND LIMITATIONS Our work is a concrete illustration of how a language model (Qwen2.5-7B-Instruct in our case) uses linear representations and additive mixing for specific model computations, number comparison in our case. The model uses linear representations for each number position despite the presence of a nonlinear manifold in its activations. We demonstrate how the model methodically combines information from both numbers by adding the number representations together, constructing a two- dimensional plane in its residual stream. The model then employs MLP neurons to focus on specific local regions within this space, which correspond to smaller intervals of the input numbers, and performs comparison on these patches. Finally, the model combines these local patches into a global comparison. In addition, we show how this algorithm extends to three number comparisons. Our findings challenge a central hypothesis in current mechanistic interpretability research, namely that detailed accounts of the representation geometry of individual concepts necessarily make it easier to characterize how models perform meaningful computations using these concepts. As we demonstrate, even though one can precisely characterize the geometry of number manifolds in model activations, simpler linear representations of numbers may be enough to causally affect the model’s computation. Our findings complement and add nuance to our understanding of what representation geometry of neural networks truly teaches us: while the intricacies of their geometric structure may reflect the properties of the underlying data distribution, simpler structures within this geometry may be sufficient computationally and useful to the model. 9 Limitations. The limitations and assumptions of our work are stated below: •Our analysis is restricted to a specific kind of input – numbers – and a specific operation – computing the maximum. Our claims about linear representations being useful despite the presence of manifold geometry hold for concepts that are ordered, i.e., those that can be mapped to the number line (without periodicity). •We rely on DAS (Geiger et al., 2024) to obtain causal directions in model activations, whenever principal components have low causal effects. Therefore, our analysis may partially inherit the same challenges as DAS, such as finding shortcuts (Wu et al., 2024). However, we include additional evidence grounded in the model, such as neurons, attention heads, etc which may partially alleviate concerns of shortcuts. •Much of our analysis shows sufficiency: we identify directions and subspaces which can modify the model’s outputs when patched. We don’t claim necessity: removing these components may not affect the model’s outputs – the model can still use other paths of information processing within its layers to solve the same task (like the hydra effect (McGrath et al., 2023)). ACKNOWLEDGMENTS This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute at Harvard University. SSRH and DB thank the Kempner Institute for access to compute resources. SSRH thanks members of the CRISP lab at Harvard SEAS for useful discussions and feedback on the manuscript. SSRH further thanks Atticus Geiger and Ekdeep Singh Lubana for insightful discussions about the project. AI USE STATEMENT In this work, we used generative AI tools for writing and editing code, literature search, drafting portions of the appendix, and feedback on research ideas and experimental methodology. We have reviewed and verified all AI-assisted work. We take responsibility for the final content of this work, including text, claims, code, or artifacts produced with the aid of generative AI. REPRODUCIBILITY STATEMENT Appendix A provides the experimental setup, prompts, datasets and sampling procedures, while Appendix B provides details of the intervention, DAS, attribution, and evaluation methods. Sample sizes and hyperparameters are summarized in Table 1. REFERENCES Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219, 2022. doi: 10.1162/colia00422. Usha Bhalla, Thomas Fel, Can Rager, Sheridan Feucht, Tal Haklay, Daniel Wurgaft, Siddharth Boppana, Matthew Kowal, Vasudev Shyam, Jack Merullo, Atticus Geiger, and Ekdeep Singh Lubana. Do sparse autoencoders capture concept manifolds?, 2026. URLhttps://arxiv. org/abs/2604.28119. Lawrence D. Brown, T. Tony Cai, and Anirban DasGupta. Interval estimation for a binomial proportion. Statistical Science, 16(2):101–133, 2001. doi: 10.1214/s/1009213286. Lawrence Chan, Adri ` a Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas.Causal scrubbing:A method for rigorously testing interpretability hypotheses.AI Align- ment Forum,https://w.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/ causal-scrubbing-a-method-for-rigorously-testing, 2022. 10 Ahmed Oumar El-Shangiti, Tatsuya Hiraoka, Hilal AlQuabeh, Benjamin Heinzerling, and Kentaro Inui. The geometry of numerical reasoning: Language models compare numeric properties in linear subspaces. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), p. 550–561, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-190-2. doi: 10.18653/v1/ 2025.naacl-short.47. URL https://aclanthology.org/2025.naacl-short.47/. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URLhttps://transformer-circuits. pub/2021/framework/index.html. Sheridan Feucht, Tal Haklay, Usha Bhalla, Daniel Wurgaft, Can Rager, Rapha ̈ el Sarfati, Jack Merullo, Thomas McGrath, Owen Lewis, Ekdeep Singh Lubana, et al. Arithmetic in the wild: Llama uses base-10 addition to reason about cyclic concepts. arXiv preprint arXiv:2605.01148, 2026. Peter G ̈ ardenfors. Conceptual spaces, volume 3. MIT press Cambridge, MA, 2000. Nikhil Garg, Jon Kleinberg, and Kenny Peng. How many features can a language model store under the linear representation hypothesis?, 2026. URLhttps://arxiv.org/abs/2602.11246. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, p. 9574–9586, 2021. arXiv:2106.02997. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D. Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Proceedings of the Third Conference on Causal Learning and Reasoning (CLeaR), volume 236 of Proceedings of Machine Learning Research, p. 160–187. PMLR, 2024. arXiv:2303.02536. Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969, 2023. Wes Gurnee and Max Tegmark. Language models represent space and time. In International Conference on Learning Representations (ICLR), 2024. arXiv:2310.02207. Wes Gurnee, Emmanuel Ameisen, Isaac Kauvar, Julius Tarng, Adam Pearce, Chris Olah, and Joshua Batson. When models manipulate manifolds: The geometry of a counting task, 2026. URL https://arxiv.org/abs/2601.04480. Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36:76033–76060, 2023. Amiri Hayes, Belinda Z Li, and Jacob Andreas. Explaining attention with program synthesis, 2026. URL https://arxiv.org/abs/2606.19317. John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 2733– 2743, 2019. doi: 10.18653/v1/D19-1275. Hyeonbin Hwang and Yeachan Park. Intrinsic task symmetry drives generalization in algorithmic tasks. arXiv preprint arXiv:2603.01968, 2026. Subhash Kantamneni and Max Tegmark. Language models use trigonometry to do addition, 2025. URL https://arxiv.org/abs/2502.00873. Dhruva Karkada, Daniel J Korchinski, Andres Nava, Matthieu Wyart, and Yasaman Bahri. Sym- metry in language statistics shapes the geometry of model representations. arXiv preprint arXiv:2602.15029, 2026. 11 J ́ anos Kram ́ ar, Tom Lieberum, Rohin Shah, and Neel Nanda. AtP*: An efficient and scalable method for localizing LLM behaviour to components. arXiv preprint arXiv:2403.00745, 2024. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity, 2024. URL https://arxiv.org/abs/2401.01967. Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2024. URLhttps://arxiv.org/abs/2310. 06824. David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010. Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations, 2023. URLhttps://arxiv. org/abs/2307.15771. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, p. 17359–17372, 2022. arXiv:2202.05262. Alexander Modell, Patrick Rubin-Delanchy, and Nick Whiteley. The origins of representation manifolds in large language models, 2025. URLhttps://arxiv.org/abs/2505.18235. Neel Nanda. Attribution patching: Activation patching at industrial scale.https://w. neelnanda.io/mechanistic-interpretability/attribution-patching , 2023. Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023. Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. In NeurIPS 2023 Workshop on Attributing Model Behavior at Scale (ATTRIB), 2023. arXiv:2310.10348. Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representations of sentiment in large language models, 2023. URLhttps://arxiv.org/abs/2310.15154. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, p. 12388–12401, 2020. Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space, 2020. URL https://arxiv.org/abs/2002.03754. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: A circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representations (ICLR), 2023. arXiv:2211.00593. Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah D. Goodman. Inter- pretability at scale: Identifying causal mechanisms in alpaca, 2024. URLhttps://arxiv. org/abs/2305.08809. Daniel Wurgaft, Can Rager, Matthew Kowal, Vasudev Shyam, Sheridan Feucht, Usha Bhalla, Tal Haklay, Eric Bigelow, Raphael Sarfati, Thomas McGrath, Owen Lewis, Jack Merullo, Noah Goodman, Thomas Fel, Atticus Geiger, and Ekdeep Singh Lubana. Manifold steering reveals the shared geometry of neural network representation and behavior, 2026. URLhttps://arxiv. org/abs/2605.05115. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. 12 Fengting Yuchi, Li Du, and Jason Eisner. Llms know more about numbers than they can say, 2026. URL https://arxiv.org/abs/2602.07812. Wentao Zhu, Zhining Zhang, and Yizhou Wang. Language models represent beliefs of self and others, 2024. URL https://arxiv.org/abs/2402.18496. 13 AEXPERIMENTAL SETUP A.1MODEL, TASK AND PROMPTS We use Qwen2.5-7B-Instruct (Yang et al., 2024) (28layers,d model = 3584,28heads of dimension 128, SwiGLU MLPs of widthd f = 18944) infloat16with all parameters frozen, on a single NVIDIA A100 40GB GPU. The taskmax(y 1 ,...,y K )is posed as a raw completion (no chat template) with a one-shot exemplar that fixes the answer format: Answer in the following format with a single answer. The maximum of 12 and 4 is 12. The maximum of y 1 and y 2 is forK = 2in both digit regimes, with the exemplar “The maximum of 12, 437 and 5 is 437.” and a question “The maximum of y 1 , y 2 and y 3 is” forK ≥ 3; each prompt ends in a single space. Qwen2.5 tokenizes numbers digit by digit, so an operand’s first token is its leading digit. We read every residual-stream quantity at a number’s last token, its position, where the whole number is first visible under causal attention; all operands are first visible at the position ofy 2 (K = 2) ory 3 (K = 3). Positions are found from the operands’ character spans after the last occurrence of"The maximum of "via the tokenizer’s offset mapping. Within an experiment all operands have the same number of digits, so every prompt has the same token layout (T = 36withy 1 ,y 2 at tokens 29, 33for two 2-digit operands;T = 38with tokens30, 35for two 3-digit operands;T = 46with tokens35, 39, 43for three 2-digit operands). All sampled operands have pairwise-distinct leading digits, so that each readout token identifies one operand. Two-digit operands lie in[10, 100)with a minimum gapg = 10between the values of a sampled tuple, and three-digit operands in[100, 1000)withg = 100; the three-digit regime is used only to replicate the number-representation analysis. A.2COUNTERFACTUAL PAIRS ForK = 2we sample triples(a,b,r)which satisfya > b + g > r + 2g(gis the gap). The clean prompt (e.g.,(a,b)) hasaat the winning operand andbat the other (answer isa), and the corrupted prompt replacesain place byr(so corrupted prompt has (r,b)) (answerb), so that only which operand wins changes. The same triples are used in both arrangements (only the order of values is changed),y 1 > y 2 andy 2 > y 1 . Patching a clean state into the corrupted run makes the model namer(smaller value in its input pair) rather thana(Section C.1): the patch changes which operand the model treats as the maximum, not the value it outputs. We therefore score interventions ont(r) againstt(b), wheret(·)is a number’s first token; unliket(a),t(r)cannot be raised by a patch that merely re-inserts the clean value. ForK = 3we sample quadruplesa > b > c > rwith consecutive gaps larger thang, placea,b,c in a chosen value order in the clean prompt, and replaceain place byrin the corrupted one, so that the corrupted winner is always the clean runner-up. The head groupsy 1 > y 2 > y 3 (patched at the position ofy 2 ),y 1 > y 3 > y 2 andy 2 > y 3 > y 1 (both patched at the position ofy 3 ) vary the runner-up’s position and the maximum’s position one at a time and are used to identify heads. All other analyses use three perturbation cases, patched at the position ofy 3 and named by the number the corruption changes: casey 1 (y 1 > y 3 > y 2 ), casey 2 (y 2 > y 3 > y 1 ) and casey 3 (y 3 > y 2 > y 1 ). A.3POPULATIONS AND SPLITS Intervention scores are computed on400evaluation examples, and directions and neuron rankings are fitted on128disjoint fitting examples, both taken from one deduplicated pool drawn by a deterministic sampler, so that every analysis uses the same examples. ForK = 3we draw1,400quadruples, keep the1,341whose clean and corrupted prompts are solved (next-token argmaxt(a)andt(b)) in all four value orders, following Meng et al. (2022), and use the first528. ForK = 2no filter is applied: in the two-digit regime the model answers every ordered pair of distinct two-digit numbers correctly, while the three-digit populations include one example per arrangement with a non-positive clean-minus-corrupted gap. Geometry and probes use separate clouds of clean prompts, with the two-number clouds containing each unordered pair in both orders. 14 Behavioural accuracy is measured by greedy decoding of one token more than the operands’ digit count, comparing the first integer of the output with the true maximum, on all8,010ordered pairs of distinct two-digit numbers and on4,000three-digit pairs drawn with replacement. The operand-count sweep uses200prompts of distinct two-digit operands per length in the three-operand prompt format, with Wilson score intervals (Brown et al., 2001). Table 1 lists all sample sizes and hyperparameters. Table 1: Sample sizes and hyperparameters. quantityvalue populations evaluation / fitting examples400 / 128 clouds (L13 two-number / L14 / three-number)2000 / 3000 / 1500 operand counts k in the accuracy sweeps, prompts per k2, 3, 4, 5, 10, 20; 200 base random seed52 learned directions Adam steps, learning rate100, 0.05 minibatch (gradients accumulated)16–32 initializations in the stability check (u)2 attribution minibatch, half-precision loss scale16, 100 candidate neurons (MLP14∪ MLP15)37,888 freeze-sweep depths k1, 3, 10, 30, 10 2 , 3·10 2 , 10 3 , 3·10 3 sweeps and fields dose-response grid values, window15, 25, . . . , 95;±2 prompts per dose-response cell (K=2)64 receptive-field gridstride 4 over y ∈ [11, 99] probes ridge penalty grid (leave-one-out)13 log-spaced values in [10 −2 , 10 4 ] held-out fraction0.2 BMETHODS B.1INTERVENTIONS AND SITES Causal measurements are interchange interventions (Geiger et al., 2021; 2024): we run the corrupted prompt and replace a component of the activation at one site by its value on the clean run. Withxthe corrupted activation,x clean the clean one and P ∈ R d×k an orthonormal basis, x ′ =x + P ⊤ x clean −x ,(4) computed in single precision.P = Iis the full-rank restoration of Meng et al. (2022). Each rank- restricted patch is shown next to a full-rank patch at the same site as a reference rather than an upper bound: rank-restricted patches can exceed it. The sites are: (i) the residual stream leaving a decoder block, at one position; (i) one attention head’s contributionz h (W h O ) ⊤ , obtained from the head’s128-dimensional slice of the output-projection input, where a directionPis patched by reading the coefficient(z clean h −z h )(W h O ) ⊤ P from the head’s contribution and adding it alongPto the residual stream (Pis not constrained to the head’s output space; forv 1 in L14.H14,81%of its norm lies inside it); (i) the pre-MLP residualx ℓ−1 + Attn ℓ (x ℓ−1 ) , patched through the attention output, which gives the same result as an end-of-layer patch at full rank; and (iv) MLP neurons, the coordinates of the post-SwiGLU vector that feeds the down-projection. B.2METRICS WithPLD = logit[t(r)]− logit[t(b)]computed at the last token, using the same token pair on the clean, corrupted and patched runs, position recovery is PR = PLD patched − PLD corr PLD clean − PLD corr ,(5) the normalized restoration of Meng et al. (2022) applied to the logit difference of Wang et al. (2023), computed per example and averaged. The numerator is the indirect effect of the patched component 15 on the PLD (Vig et al., 2020), and the denominator is a per-example logit scale of10–13logits on average, which makes values comparable across sites and cases but is not meant to make a full effect equal1: sincet(r)is not an answer on the clean prompt,PLD clean can be negative. Where IIA is near zero, as in three-number casesy 1 andy 2 , PR measures how far the PLD moves rather than a change of answer. PR need not lie in [0, 1]. Interchange intervention accuracy (Geiger et al., 2021) isIIA = I [arg max v∈V logit[v] = t(r)]over the full vocabulary at the last token, which scores the first generated token, the leading digit ofr, so it also counts the mixed answers produced by a full-rank patch at the last token ofy 2 (Section C.1). For the ablations we use full-number accuracy: the answer is decoded greedily for one token more than the operands’ digit count and counted correct when its leading characters match the true maximum. B.3DIRECTIONS Learned directions are rank-ksubspaces found by distributed alignment search for a single causal variable (Geiger et al., 2024): the basisQ = qr(R)(using QR factorization) of a free matrixRis inserted into Equation 4 at the chosen site and trained on the fitting examples with a cross-entropy towardt(r)restricted tot(b),t(r), with gradients accumulated so that each step uses the full-batch mean. With 16 fitting examples a fit reached a training IIA of 1.000 against a held-out 0.688, hence the128used here. Retrainingufrom a second initialization gives| cos| = 0.99and the same held-out IIA (0.94) for two-digit operands, and| cos| = 0.82with IIAs0.86and0.80for three-digit operands; other directions use a single initialization. Where a site may carry one variable shared across cases, a subspace fitted on each case is evaluated on every case. Directions are signed so that their component rises with the number they carry. Table 2 lists the directions. Table 2: Directions, where they live and how they are obtained. DAS directions are rank one and fitted in the listed case; PC denotes the top principal component(s) of the corresponding clean cloud. Kdirectionpositionsiteobtained by 2uy 1 L13 residualDAS, y 1 perturbed 2v 1 y 2 output of L14.H14DAS, y 1 perturbed 2v 2 y 2 L13 residualPC 2comparison flagy 2 L15 residualDAS, each case 3u 1 ,u 2 y 1 , y 2 L13 residualDAS, cases y 1 , y 2 3v 1 ,v 2 y 3 outputs of L14.H14, L14.H18DAS, cases y 1 , y 2 3v 3 y 3 L13 residualPC 3w 1 ,w 2 y 2 L13 residualDAS, cases y 1 , y 2 (w 2 =u 2 ) 3comparison flags y 2 , y 3 L15 residualDAS, each case 3answer subspacelast tokenL20 residualtop two PCs v 2 is a principal component because a direction learned at that site partly encodes the comparison outcome instead ofy 2 (Section C.3), and heads L14.H14 and L14.H18 come from our earlier local- ization of the three-number task. Multi-dimensional patches use the span of thev j , orthogonalized by Gram–Schmidt withv 1 first; the directions are nearly orthogonal to begin with (cos(v 1 ,v 2 ) =−0.08 for two numbers), and because they are obtained independently, the span’s score is a lower bound for a rank-k patch at that site. Principal components are fitted on mean-centred, unscaled activations (Gurnee & Tegmark, 2024). Linear decodability is measured with ridge regression (Alain & Bengio, 2016; Belinkov, 2022) on the logarithm of each number, and the comparisony 1 > y 2 with a ridge classifier, reporting held-out R 2 or accuracy; probes for operands that a position cannot depend on serve as floors, in the spirit of the control tasks of Hewitt & Liang (2019). Log-linear trends are summarized by the least-squares fit p logy + q. B.4LOCALIZATION The causal trace (Meng et al., 2022) restores the clean residual stream leaving one layer at one token, for every layer and every token from the first token ofy 1 to the last token of the prompt, and reports position recovery averaged within each arrangement. Heads are then interchanged at site (i) of layer 14 (Wang et al., 2023; Goldowsky-Dill et al., 2023), each together with a restoration of the layer-13 16 residual at the same position (in full for three numbers, alongv 2 for two numbers), since a single head written into a corrupted residual stream has almost no effect; the co-patch alone is shown as the baseline, and for two numbers the sweep is also shown without it. At the last token, full-rank patches of the residual leaving layers18to22over the head-group orders locate the read-out at layer 20, where mean position recovery increases most (0.09to0.55). At that layer we patch each head’s contribution without a co-patch, with the whole attention sublayer as a reference, and compare a patch of the top two principal components of the residual with a full-rank patch. B.5NEURONS We rank the neurons of MLP14 and MLP15 by attribution patching (Nanda, 2023; Syed et al., 2023; Kram ́ ar et al., 2024), the first-order estimate of the effect on the PLD of freezing neuroniwhile the shared representation is patched,s i ≈ (a corr i − a patched i )∂PLD/∂a i , with the gradient taken at the patched state through a zero-valued differentiable input at the down-projection. Rankings are computed on the fitting examples and verified by freezing the topkneurons on the evaluation examples, against two random draws ofkneurons. The shared set is the intersection of the top-20 rankings of the two two-number cases,12neurons (six in each MLP), which the three-number analysis reuses unchanged. To freeze a sub-block or a set of neurons, we pin its output (for neurons, their post-SwiGLU values) at the target position to its value on the unpatched corrupted run, the severed-path variant of causal tracing (Meng et al., 2022; Vig et al., 2020), for the attention and MLP sub-blocks of layers14to 16individually and for MLP14 and MLP15 jointly. For injection we write only the clean outputs of the selected MLPs into the corrupted run at the target position. For ablation we set the shared neurons’ post-SwiGLU activations to zero, at every position or only at the last token of the final operand, throughout greedy decoding, and compare with two random sets of12neurons zeroed at every position, scoring full-number accuracy. B.6DOSE-RESPONSE SWEEPS, RECEPTIVE FIELDS AND CONNECTIVITY For the dose-response sweep (Geiger et al., 2021; Chan et al., 2022) we set the two coordinates of the(v 1 ,v 2 )frame in the pre-MLP layer-14 residual at the position ofy 2 to the mean coordinate of a chosen number (over a±2window in the cloud), on a9× 9grid with one value per leading digit, and score the fraction of argmax answers that name the first operand among those naming either operand, masking the diagonal. The pre-MLP site is used because a per-number mean explainsR 2 = 0.87of thev 1 coordinate there, against 0.35 at the end of the layer. A neuron’s receptive field is its post-SwiGLU activation at the target position over a grid of prompts, one forward pass per cell and without the distinct-leading-digit constraint. Three-number fields are shown as pairwise marginals, each averaged over the third number and divided by its own peak, and an operand that a position cannot depend on is held fixed (y 3 = 55 at the position of y 2 ). The causal edge from an MLP14 neuron to an MLP15 neuron is the shift in the MLP15 neuron’s activation when only the MLP14 neuron is set to its clean value, in units of the MLP15 neuron’s standard deviation over a1,500-prompt two-number cloud. The virtual weight (Elhage et al., 2021) through the gate projection is the cosine between the upstream write column and the downstream gate read row with the RMSNorm scale folded in, and the alignment of a write directionw j with a learned subspaceQisρ j =∥Q ⊤ w j ∥/∥w j ∥, read against the distribution over all18,944neurons of the layer, since write directions are not isotropic (the analytic level for a random unit vector and a k-dimensional subspace is about p k/d model ). As a specificity control, each three-number direction is re-evaluated, without refitting, in the cases where the number it carries does not change, where it should have no effect. 17 CFURTHER EXPERIMENTAL RESULTS The figures below follow the order of the main-text algorithm. Unless a caption says otherwise, scores are means over the400held-out counterfactuals of Section A.3, error bars are one standard error, and PR (position recovery) and IIA (interchange intervention accuracy) are as defined in Section B.2. C.1BEHAVIOUR AND WHAT AN INTERVENTION CHANGES We first report task accuracy and the answers the model generates under two interventions. 23451020 number of operands, k 0.75 0.80 0.85 0.90 0.95 1.00 accuracy Figure 6: Accuracy onmax(y 1 ,...,y k )against the number of operandsk, under greedy decoding, for 200 prompts of distinct two-digit operands per length. Error bars are Wilson score intervals. 0.00.20.40.60.81.0 fraction of generated answers no patch u at y 1 full at y 2 r b a leading digit of r, last digit of a other Figure 7: Answers generated by greedy decoding on the400held-out counterfactuals, as fractions per category:r,b,a, the leading digit ofrfollowed by the last digit ofa, and other. Rows: the corrupted prompt with no patch and with the full layer-13 residual patched at the last token ofy 2 (both withy 2 perturbed), and withupatched at the position ofy 1 (y 1 perturbed). An answer equal toris counted as r. Table 3: Examples of generated answers on held-out counterfactuals: the clean and corrupted operands, their maxima, and the answer generated under each intervention. interventionclean (y 1 , y 2 )maxcorrupted (y 1 , y 2 )maxpatched answer u at y 1 (99, 61)99(12, 61)6112 u at y 1 (70, 55)70(33, 55)5533 full at y 2 (61, 99)99(61, 12)6112 full at y 2 (56, 97)97(56, 27)5627 full at y 2 (55, 70)70(55, 33)5530 full at y 2 (71, 89)89(71, 52)7159 18 C.2WHERE THE COMPUTATION HAPPENS Causal traces and linear probes over every layer and question token (Section B.4) locate the sites analysed in the rest of the appendix. y 1 y 2 last token, y 1 perturbed 0 3 6 9 12 15 18 21 24 27 layer (a) y 1 y 2 last token, y 2 perturbed (b) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 PR Figure 8: Position recovery when the clean residual stream leaving one layer (rows) is restored at one token (columns), from the first token ofy 1 to the last token of the prompt, with (a)y 1 and (b)y 2 perturbed. Each operand spans two tokens and the tick marks its last one. The panels share a colour scale. The maximum 3 and 10 is of ⋯ Layer 13Layer 14Layer 15 Residual u Residual v₂ H14: attends, copies and transforms H14 output + v₁ Specific MLP neurons Argmax Specific MLP neurons Figure 9: Illustration of the key components implementing algorithm underlying the pairwise number comparisons algorithm Alg. 1 19 y 1 y 2 last token 0 3 6 9 12 15 18 21 24 27 layer (a) y 1 y 2 last token (b) y 1 y 2 last token (c) 0.0 0.2 0.4 0.6 0.8 1.0 R 2 , log y 1 0.0 0.2 0.4 0.6 0.8 1.0 R 2 , log y 2 0.5 0.6 0.7 0.8 0.9 1.0 accuracy, y 1 > y 2 Figure 10: Ridge probes fitted at every layer (rows) and token (columns) on2,000clean two-number prompts and scored on a20%held-out split. (a,b) Held-outR 2 forlogy 1 andlogy 2 . (c) Held-out accuracy of a ridge classifier for y 1 > y 2 . The axes are those of Figure 8. y 1 y 2 y 3 last token, y 1 perturbed 0 3 6 9 12 15 18 21 24 27 layer (a) y 1 y 2 y 3 last token, y 2 perturbed (b) y 1 y 2 y 3 last token, y 3 perturbed (c) 0.0 0.2 0.4 0.6 0.8 1.0 PR Figure 11: The causal trace of Figure 8 for three numbers, with (a)y 1 , (b)y 2 and (c)y 3 perturbed, on the 400 held-out quadruples. The panels share a colour scale. 20 C.3INDIVIDUAL NUMBER REPRESENTATIONS For the directionuat the position ofy 1 , we show its place in the layer-13 geometry, a comparison with the top principal component, the same analysis at earlier layers and with three-digit operands. We then show the learned alternative to v 2 and the output of the transport head L14.H14. −20−10010 PC1 −20 −15 −10 −5 0 5 10 PC3 (a) 20406080100 y 1 −20 −15 −10 −5 0 5 10 component along u (b) 10 20 30 40 50 60 70 80 90 y 1 Figure 12: The layer-13 residual at the position ofy 1 , one point per value ofy 1 . (a) Projection on the first and third principal components, coloured byy 1 , with the smoothed mean position alongy 1 (line) and the direction ofu’s projection onto the plane (arrow). (b) Component alonguagainsty 1 , with the least-squares fit p logy 1 + q (dotted). full u PC1 0.0 0.2 0.4 0.6 0.8 IIA (a) full u PC1 0.0 0.5 1.0 1.5 2.0 PR (b) 20406080100 y 1 −2 −1 0 1 standardized component (c) u PC1 15101520 principal component 0.000 0.025 0.050 0.075 0.100 0.125 0.150 variance explained (d) u Figure 13: The layer-13 residual at the position ofy 1 , withy 1 perturbed. (a) IIA and (b) position recovery of a full-rank patch, a rank-one patch ofuand a rank-one patch of the top principal component (PC1). (c) Standardized components alonguand PC1 againsty 1 , as per-value means. (d) Variance explained by each of the first 20 principal components (bars) and by u (dashed line). 21 135791113 layer, patched at y 1 0.0 0.2 0.4 0.6 0.8 IIA (a) full u PC1 135791113 layer, patched at y 2 0.2 0.4 0.6 0.8 1.0 IIA (b) full v 2 20406080100 y 1 −7.5 −5.0 −2.5 0.0 2.5 5.0 component along u , L3 (c) 20406080100 y 1 −15 −10 −5 0 5 10 component along u , L7 (d) 20406080100 y 1 −20 −15 −10 −5 0 5 10 component along u , L13 (e) Figure 14: The number-representation analysis at layers1to13, withurefitted at each layer. (a) IIA at the position ofy 1 , withy 1 perturbed, for a full-rank patch,uand that layer’s top principal component. (b) IIA at the position ofy 2 , withy 2 perturbed, for a full-rank patch andv 2 , taken as that layer’s top principal component. (c–e) Component alonguagainsty 1 at layers 3, 7 and 13, as per-value means, with the fit p logy 1 + q (dotted). 2004006008001000 y 1 −10 −5 0 5 component along u (a) 2004006008001000 y 2 −20 −10 0 10 component along v 2 (b) full u PC1full v 2 0.0 0.2 0.4 0.6 0.8 1.0 IIA y 1 y 2 (c) Figure 15: The number representations with three-digit operands. (a) Component alonguat the position ofy 1 againsty 1 , and (b) component alongv 2 at the position ofy 2 againsty 2 , both averaged in bins of25, with the fitp logy + q(dotted). (c) IIA of a full-rank patch,uand PC1 at the position of y 1 with y 1 perturbed, and of a full-rank patch and v 2 at the position of y 2 with y 2 perturbed. 20406080100 y 2 −10 −5 0 5 10 DAS component at y 2 (a) y 1 > y 2 y 2 > y 1 20406080100 y 2 −20 −10 0 10 20 component along v 2 (b) fullDAS v 2 0.0 0.2 0.4 0.6 0.8 1.0 IIA, y 2 perturbed (c) Figure 16: The layer-13 residual at the position ofy 2 on the2,000-prompt cloud, coloured by which number is larger. (a) Component along a rank-one direction fitted withy 2 perturbed, againsty 2 . (b) Component alongv 2 , the top principal component, againsty 2 . (c) IIA withy 2 perturbed for a full-rank patch, the fitted direction (DAS) and v 2 . 22 −6−4−2024 PC1 of H14 output −6 −4 −2 0 2 4 6 PC2 of H14 output (a) v 1 20406080100 y 1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 H14 attention to y 1 (b) 20406080100 y 1 −4 −2 0 2 component along v 1 (c) 10 20 30 40 50 60 70 80 90 y 1 Figure 17: The output of head L14.H14 at the position ofy 2 . (a) Its top two principal components, coloured byy 1 , with the smoothed mean position alongy 1 (line) and the direction ofv 1 ’s projection onto the plane (arrow). (b) The head’s attention from the position ofy 2 toy 1 , againsty 1 , as per-value means. (c) Component along v 1 against y 1 , as per-value means. 23 C.4THE SHARED REPRESENTATION Probes and position recovery complement the main-text IIA results for the shared representation at the position of y 2 . 20406080100 −6 −4 −2 0 2 component along p 1 (a) 20406080100 (b) 20406080100 y 1 −10 −5 0 5 component along p 2 (c) 20406080100 y 2 (d) Figure 18: Components of the pre-MLP layer-14 residual at the position ofy 2 along ridge probes forlogy 1 (p 1 , top row) andlogy 2 (p 2 , bottom row), againsty 1 (left) andy 2 (right), on600held-out prompts. L13 full v 2 v 1 v 1 & v 2 v 1 , v 2 plane L14 full patched component 0.0 0.5 1.0 1.5 2.0 PR y 1 perturbedy 2 perturbed Figure 19: Position recovery at the position ofy 2 for the patches of the main-text shared-representation figure, withy 1 ory 2 perturbed: the full layer-13 residual,v 2 ,v 1 ,v 1 andv 2 as two separate rank-one patches, the rank-two(v 1 , v 2 )plane in the pre-MLP layer-14 residual, and the full pre-MLP layer-14 residual. Hatched bars are the two conditions that patch both directions. 24 C.5THE COMPARATOR NEURONS Freezing, attribution, receptive fields and connectivity for MLP14 and MLP15 appear in the order of the selection procedure of Sections B.5–B.6. none MLP14MLP15MLP16 attn15attn16 MLP14+15 frozen sub-block 0.0 0.2 0.4 0.6 0.8 1.0 IIA (a) y 1 perturbedy 2 perturbed none MLP14 MLP14, 15 MLP14, 15, 16 injected MLP outputs 0.0 0.2 0.4 0.6 0.8 1.0 IIA (b) Figure 20: (a) IIA of the rank-two(v 1 , v 2 )plane patch when one sub-block’s output at the position ofy 2 is frozen to its corrupted value, withy 1 ory 2 perturbed. The layer-14 attention freeze is omitted, because the pre-MLP patch is written through that output (Section B.1). (b) IIA when only the clean outputs of the listed MLPs are injected at the position of y 2 . −10 0 −10 −1 0 10 −1 10 0 attribution, y 1 perturbed −10 0 −10 −1 0 10 −1 10 0 attribution, y 2 perturbed (a) all neurons shared top 20 10 0 10 1 10 2 10 3 neurons frozen, k 0.0 0.2 0.4 0.6 0.8 IIA (b) y 1 perturbed y 2 perturbed top k random k none shared 12 own top 20 random 12 frozen neurons 0.0 0.2 0.4 0.6 0.8 1.0 IIA (c) Figure 21: (a) Attribution scores of the37,888neurons of MLP14 and MLP15, withy 1 perturbed (x-axis) andy 2 perturbed (y-axis), on symmetric-log axes. The12neurons in the top20of both cases are highlighted. (b) IIA of the plane patch when the topkneurons by attribution (solid) orkrandom neurons (dashed) are frozen. (c) IIA with no neurons frozen, with the12shared neurons frozen, with each case’s own 20 top-ranked neurons frozen, and with 12 random neurons frozen. 20 50 80 y 2 L14 #6262L14 #6150L14 #5076L14 #9459L14 #6854L14 #15775 205080 y 1 20 50 80 y 2 L15 #12784 205080 y 1 L15 #3003 205080 y 1 L15 #8383 205080 y 1 L15 #3616 205080 y 1 L15 #9442 205080 y 1 L15 #11556 −1.00 −0.75 −0.50 −0.25 0.00 0.25 0.50 0.75 1.00 activation / peak Figure 22: Receptive fields of the12shared neurons, six in MLP14 (top) and six in MLP15 (bottom), ordered by worst-case attribution rank. Each field is the post-SwiGLU activation at the position ofy 2 over a grid of (y 1 ,y 2 ) prompts with stride 4, divided by its peak. The dashed line is y 1 = y 2 . 25 20 50 80 y 2 L15 #11605 y 1 perturbed L14 #4724L15 #5205L14 #9341L15 #6448L15 #14710 205080 y 1 20 50 80 y 2 L15 #5795 y 2 perturbed 205080 y 1 L14 #5222 205080 y 1 L15 #7514 205080 y 1 L14 #682 205080 y 1 L15 #188 205080 y 1 L14 #11120 −1.00 −0.75 −0.50 −0.25 0.00 0.25 0.50 0.75 1.00 activation / peak Figure 23: Receptive fields, drawn as in Figure 22, of the six highest-ranked neurons that are in the top 20 of one perturbation case only: y 1 perturbed (top) and y 2 perturbed (bottom). #3003#3616#8383#9442 #11556#12784 MLP15 neuron #5076 #6150 #6262 #6854 #9459 #15775 MLP14 neuron (a) #3003#3616#8383#9442 #11556#12784 MLP15 neuron #5076 #6150 #6262 #6854 #9459 #15775 (b) #3003#3616#8383#9442 #11556#12784 MLP15 neuron #5076 #6150 #6262 #6854 #9459 #15775 (c) −0.4 −0.2 0.0 0.2 0.4 causal edge (SD), y 1 perturbed −0.4 −0.2 0.0 0.2 0.4 causal edge (SD), y 2 perturbed −0.2 −0.1 0.0 0.1 0.2 virtual weight (gate) Figure 24: (a,b) Causal edges from each shared MLP14 neuron (rows) to each shared MLP15 neuron (columns): the shift in the MLP15 neuron’s activation, in units of its standard deviation, when only the MLP14 neuron is set to its clean value, with (a)y 1 or (b)y 2 perturbed. (c) Virtual weights through the gate projection, computed as the cosine between each MLP14 neuron’s write direction and each MLP15 neuron’s normalization-folded gate read direction (Section B.6). y 1 perturbedy 2 perturbed evaluated with 0.0 0.2 0.4 0.6 0.8 1.0 IIA (a) fitted, same case fitted, other case full rank 0.000.050.100.150.20 ρ j , direction fitted with y 1 perturbed 10 0 10 1 10 2 10 3 MLP15 neurons (b) 0.000.050.100.150.200.25 ρ j , direction fitted with y 2 perturbed 10 0 10 1 10 2 10 3 (c) Figure 25: (a) IIA of a rank-one direction in the layer-15 residual at the position ofy 2 , fitted in the case it is evaluated on (fitted, same case) or in the other case (fitted, other case), and of the full-rank layer-15 patch, for each perturbation case. (b,c) Histograms of the alignmentρ j of the write direction of every MLP15 neuron with the direction fitted with (b)y 1 or (c)y 2 perturbed, on a log count axis. Vertical lines mark the six shared MLP15 neurons. 26 C.6THREE NUMBERS A summary figure covers the positions ofy 3 andy 2 and the last token, and the figures after it expand each part. u 1 u 2 v 1 v 2 v 3 u 1 u 2 v 1 v 2 v 3 +1.00 +0.70+1.00 -0.16-0.17+1.00 +0.04+0.08+0.09+1.00 +0.49+0.49-0.04-0.00+1.00 v 1 v 2 v 3 v 1 & v 2 & v 3 v 1 , v 2 , v 3 space full 0.0 0.2 0.4 0.6 0.8 1.0 PR y 1 pert. y 2 pert. y 3 pert. 255075 y 1 20 40 60 80 y 2 local L14#5076 255075 y 1 20 40 60 80 y 3 255075 y 2 20 40 60 80 y 3 255075 y 1 20 40 60 80 y 2 global L15#12784 255075 y 1 20 40 60 80 y 3 255075 y 2 20 40 60 80 y 3 105051015 L15 DAS k=1 (fit y 3 ) 0 20 40 60 80 100 prompts DASfull 0.0 0.5 1.0 1.5 2.0 PR y 3 not max y 3 max u 1 u 2 w 1 w 2 u 1 u 2 w 1 w 2 +1.00 +0.70+1.00 +0.39+0.35+1.00 +0.70+1.00+0.35+1.00 w 1 w 2 w 1 & w 2 w 1 , w 2 plane full 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 PR y 1 pert. y 2 pert. 255075 y 1 20 40 60 80 y 2 local L14#5076 255075 y 1 20 40 60 80 y 2 global L15#12784 105051015 L15 DAS k=1 (fit y 2 ) 0 20 40 60 80 prompts DASfull 0 1 2 3 PR y 2 not max y 2 max 201001020 PC1 20 10 0 10 PC2 max y 1 max y 2 max y 3 y 1 y 2 y 3 is token position 0.0 0.2 0.4 0.6 0.8 H27 attention max y 1 max y 2 max y 3 H27 attn. heads (L20) 0.00 0.05 0.10 0.15 PR y 1 > y 2 y 1 > y 3 y 2 > y 3 0.0 0.2 0.4 0.6 0.8 PR full-rank 2D PCA (a)(b)(c)(d) (e)(f) (g) (h) (i) (j) (k)(l) y 3 position y 2 position last token Figure 26: The three-number analysis at the position ofy 3 (top row), the position ofy 2 (middle row) and the last token (bottom row). (a) Cosine similarities betweenu 1 ,u 2 ,v 1 ,v 2 andv 3 ; the pairs amongv 1 , v 2 , v 3 are boxed. (b) Position recovery at the position ofy 3 , by perturbed number, for each direction alone, the three as separate rank-one patches, their span in the pre-MLP layer-14 residual, and the full layer-14 residual. (c) Receptive fields of L14#5076 (top) and L15#12784 (bottom) at the position ofy 3 over a(y 1 ,y 2 ,y 3 )grid with stride4, shown as the three pairwise marginals, each averaged over the third number and divided by its own peak. (d) The1,500-prompt cloud projected on a rank-one direction in the layer-15 residual fitted withy 3 perturbed, split by whethery 3 is the maximum; the inset shows the position recovery of this direction and of the full layer-15 residual. (e–h) The same at the position ofy 2 , wherew 1 andw 2 are rank-one directions in the layer-13 residual fitted withy 1 andy 2 perturbed (w 2 andu 2 are the same fit). The fields in (g) are over(y 1 ,y 2 )with y 3 = 55. (i) Top two principal components of the residual leaving layer 20 at the last token, coloured by which operand holds the maximum. (j) Attention of head L20.H27 from the last token to the question tokens, averaged by the position of the maximum. (k) Position recovery of each layer-20 head’s own output at the last token, for the ordery 1 > y 2 > y 3 . (l) Position recovery of a full-rank patch and of a patch of the top two principal components of the layer-20 residual at the last token, for three value orders; the principal components are fitted on a separate 1,500-prompt cloud. 27 20406080100 y 1 −10 −5 0 5 10 15 component along u 1 (a) 20406080100 y 2 −10 −5 0 5 10 component along u 2 (b) 20406080100 y 1 −2 −1 0 1 2 component along v 1 (c) 20406080100 y 2 −2 −1 0 1 2 component along v 2 (d) 20406080100 y 3 −20 −10 0 10 component along v 3 (e) Figure 27: Component along each direction of the three-number analysis against the number it carries, as per-value means on the1,500-prompt cloud. (a,b)u 1 andu 2 in the layer-13 residual at the positions ofy 1 andy 2 . (c,d)v 1 andv 2 in the outputs of heads L14.H14 and L14.H18 at the position of y 3 . (e) v 3 in the layer-13 residual at the position of y 3 . −4−2024 PC1 of H14 output −4 −2 0 2 4 PC2 of H14 output (a) −4−2024 PC1 of H18 output −4 −2 0 2 4 PC2 of H18 output (b) −20−10010 PC1 of L13 residual −20 −15 −10 −5 0 5 10 15 PC2 of L13 residual (c) 10 20 30 40 50 60 70 80 90 y 1 10 20 30 40 50 60 70 80 90 y 2 10 20 30 40 50 60 70 80 90 y 3 Figure 28: The spaces at the position ofy 3 that contain the three-number directions, each in its top two principal components, coloured by the number its direction carries, with the smoothed mean position along that number (line) and the direction of the corresponding projection onto the plane (arrow). (a) The output of head L14.H14 withv 1 . (b) The output of head L14.H18 withv 2 . (c) The layer-13 residual with v 3 . y 1 y 2 y 3 perturbed number u 1 u 2 v 1 v 2 v 3 patched direction 2.050.000.00 -0.013.080.00 0.59-0.010.00 0.010.210.00 -0.03-0.060.83 0.0 0.2 0.4 0.6 0.8 1.0 PR Figure 29: Position recovery of a rank-one patch of each direction (rows) in each perturbation case (columns). Colours are clipped at 1; printed values are not. 28 H14H11H16H13H20H2H6H26 layer-14 head −0.10 −0.05 0.00 0.05 0.10 0.15 PR y 1 > y 2 > y 3 , at y 2 y 1 > y 3 > y 2 , at y 3 y 2 > y 3 > y 1 , at y 3 Figure 30: Position recovery of each layer-14 head’s own output, interchanged at the runner-up’s position together with the full layer-13 residual there, for three value orders and the eight heads with the largest mean absolute effect. Dashed lines show the layer-13 patch alone for each order. none MLP14MLP15MLP16 attn15attn16 MLP14+15 frozen sub-block 0.0 0.2 0.4 0.6 0.8 PR (a) y 1 perturbedy 2 perturbedy 3 perturbed 10 0 10 1 10 2 10 3 neurons frozen, k 0.0 0.2 0.4 0.6 0.8 PR (b) top k random k y 1 perturbed y 2 perturbed y 3 perturbed 0.0 0.2 0.4 0.6 0.8 PR (c) none frozen two-number 12 random 12 Figure 31: Position recovery of the patch of the span ofv 1 , v 2 , v 3 in the pre-MLP layer-14 residual at the position ofy 3 , by perturbed number, while parts of the network are frozen. (a) One sub-block’s output at the position ofy 3 frozen to its corrupted value. (b) The topkneurons by this task’s attribution ranking (solid) orkrandom neurons (dashed) frozen. (c) No neurons frozen, the12shared neurons of the two-number task frozen, and 12 random neurons frozen. y 1 y 2 y 3 perturbed number, at y 3 y 1 y 2 y 3 full rank direction fitted with 1.870.290.08 0.962.330.07 0.140.032.02 0.350.151.02 (a) y 1 y 2 perturbed number, at y 2 y 1 y 2 full rank 1.170.02 -0.023.20 -0.040.20 (b) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 PR 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 PR Figure 32: Position recovery of a rank-one direction in the layer-15 residual fitted in one case (rows) and evaluated in each case (columns), with the full-rank layer-15 patch in the last row, (a) at the position of y 3 and (b) at the position of y 2 . Colours are clipped at 2; printed values are not. −10−505 PC1, L18 −5 0 5 10 PC2 (a) −10010 PC1, L19 −10 −5 0 5 10 15 (b) −20020 PC1, L20 −20 −10 0 10 (c) −20020 PC1, L21 −10 0 10 20 (d) −40−20020 PC1, L22 −20 0 20 (e) max y 1 max y 2 max y 3 Figure 33: Top two principal components of the residual stream leaving layers 18 to 22 at the last token, on 1,500 clean three-number prompts, coloured by which operand holds the maximum. 29 1819202122 layer (last token) 0.0 0.2 0.4 0.6 0.8 1.0 PR (a) y 1 > y 2 > y 3 y 1 > y 3 > y 2 y 2 > y 3 > y 1 H27H26H0H1H24H17H22H6 layer-20 head (last token) 0.00 0.05 0.10 0.15 0.20 PR (b) Figure 34: (a) Position recovery of a full-rank patch of the residual leaving each of layers 18 to 22 at the last token, for three value orders. (b) Position recovery of each layer-20 head’s own output at the last token, for the eight heads with the largest effect and the same three orders. Dashed lines show a patch of the whole layer-20 attention sublayer for each order (Section B.4). 23451020 number of operands, k 0.80 0.85 0.90 0.95 1.00 accuracy intact 12 zeroed, all tokens 12 zeroed, last token random 12 zeroed Figure 35: Accuracy onmax(y 1 ,...,y k )for200two-digit prompts per length, scored on the full decoded number (Section B.2): the intact model, the12shared neurons zeroed at every token, the same neurons zeroed only at the last token of the final operand (“last token” in the legend), and12 random neurons zeroed at every token (two random sets, pooled). 30 C.7IIA ON THE WHOLE GENERATED NUMBER We rescore the main-text interventions on the whole number the model generates. y 1 perturbedy 2 perturbed 0.0 0.2 0.4 0.6 0.8 1.0 IIA, first digit (a) y 1 perturbedy 2 perturbed 0.0 0.2 0.4 0.6 0.8 1.0 IIA, whole number (b) full u v 1 v 1 , pre-MLP v 2 Figure 36: IIA of the number-representation patches, scored on (a) the first generated token, the leading digit ofr(Section B.2), and (b) the whole generated number: the answer decoded greedily for one token more than the operands’ digit count, counted correct when its first integer equalsr. Withy 1 perturbed: the full layer-13 residual anduat the position ofy 1 , andv 1 at the position of y 2 , patched inside the output of L14.H14 as in the main text or in the pre-MLP layer-14 residual (hatched). Withy 2 perturbed: the full layer-13 residual andv 2 at the position ofy 2 . The rank-one patches score the same on both, while the full-rank patches mostly generate a number other thanr that starts with its leading digit. L13 full v 2 v 1 v 1 & v 2 v 1 , v 2 plane L14 full patched component 0.0 0.2 0.4 0.6 0.8 1.0 IIA, first digit (a) L13 full v 2 v 1 v 1 & v 2 v 1 , v 2 plane L14 full patched component 0.0 0.2 0.4 0.6 0.8 1.0 IIA, whole number (b) y 1 perturbedy 2 perturbed Figure 37: IIA of the patches of the main-text shared-representation figure at the position ofy 2 , with y 1 ory 2 perturbed, scored on (a) the first generated token and (b) the whole generated number, as in Figure 36: the full layer-13 residual,v 2 ,v 1 ,v 1 andv 2 as two separate rank-one patches, the rank-two(v 1 , v 2 )plane in the pre-MLP layer-14 residual, and the full residual leaving layer 14. Hatched bars are the two conditions that patch both directions. Only the full-rank patches withy 2 perturbed lose IIA on the whole number. 31 first digitwhole number 0.0 0.2 0.4 0.6 0.8 1.0 IIA, DAS 1D at L15 y 1 perturbedy 2 perturbed Figure 38: IIA of the rank-one direction in the layer-15 residual at the position ofy 2 , fitted withy 1 perturbed and evaluated in both cases, scored on the first generated token and on the whole generated number, as in Figure 36. Table 4: IIA on the first generated token and on the whole generated number for the patches of Figures 36–38. Dashes mark cases in which a patch is not evaluated. y 1 perturbedy 2 perturbed patchfirst digitwhole numberfirst digitwhole number L13 full at y 1 0.0720.003– u0.9400.940– v 1 0.4600.4600.0000.000 v 1 , pre-MLP0.7100.710– L13 full at y 2 0.0000.0000.9800.115 v 2 0.0000.0000.7900.785 v 1 &v 2 0.6750.6750.8850.880 (v 1 ,v 2 ) plane0.8100.8100.9480.940 L14 full0.8250.8250.9850.115 L15 DAS1.0001.0000.6550.650 32 Table 5: Generated answers under the patches of Figure 36, five per patch, chosen to include correct answers and mistakes: the perturbed number, the clean and corrupted operands,r, and the generated number. A patch succeeds when the model generates r. patchperturbedclean (y 1 , y 2 )corrupted (y 1 , y 2 )rgenerated L13 full at y 1 y 1 (93, 41)(13, 41)1313 y 1 (96, 46)(20, 46)2026 y 1 (69, 42)(13, 42)1312 y 1 (60, 38)(15, 38)1512 y 1 (99, 61)(12, 61)1261 uy 1 (99, 61)(12, 61)1212 y 1 (70, 55)(33, 55)3333 y 1 (89, 71)(52, 71)5252 y 1 (95, 64)(27, 64)2727 y 1 (52, 33)(19, 33)1933 v 1 y 1 (99, 61)(12, 61)1212 y 1 (95, 64)(27, 64)2727 y 1 (75, 49)(24, 49)2424 y 1 (96, 46)(20, 46)2020 y 1 (70, 55)(33, 55)3355 v 1 , pre-MLPy 1 (99, 61)(12, 61)1212 y 1 (70, 55)(33, 55)3333 y 1 (95, 64)(27, 64)2727 y 1 (87, 60)(34, 60)3434 y 1 (89, 71)(52, 71)5271 L13 full at y 2 y 2 (61, 99)(61, 12)1212 y 2 (56, 97)(56, 27)2727 y 2 (55, 70)(55, 33)3330 y 2 (71, 89)(71, 52)5259 y 2 (33, 52)(33, 19)1952 v 2 y 2 (61, 99)(61, 12)1212 y 2 (71, 89)(71, 52)5252 y 2 (44, 90)(44, 11)11111 y 2 (45, 79)(45, 11)11111 y 2 (55, 70)(55, 33)3355 33 Table 6: Generated answers, as in Table 5, under the remaining patches of Figures 37 and 38. patchperturbedclean (y 1 , y 2 )corrupted (y 1 , y 2 )rgenerated v 1 &v 2 y 1 (99, 61)(12, 61)1212 y 2 (61, 99)(61, 12)1212 y 2 (44, 90)(44, 11)11111 y 2 (45, 79)(45, 11)11111 y 2 (55, 70)(55, 33)3355 (v 1 ,v 2 ) plane y 1 (99, 61)(12, 61)1212 y 2 (61, 99)(61, 12)1212 y 2 (62, 94)(62, 11)11111 y 2 (56, 88)(56, 11)11111 y 1 (89, 71)(52, 71)5271 L14 fully 2 (61, 99)(61, 12)1212 y 1 (70, 55)(33, 55)3333 y 2 (55, 70)(55, 33)3330 y 2 (71, 89)(71, 52)5259 y 1 (99, 61)(12, 61)1261 L15 DASy 1 (99, 61)(12, 61)1212 y 1 (70, 55)(33, 55)3333 y 2 (62, 75)(62, 10)10102 y 2 (55, 83)(55, 10)10105 y 2 (61, 99)(61, 12)1261 34