Paper deep dive

Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1 8B

Rahul Chowdhury, Timothy A Rupprecht, Senhao Cao, Jiahao Liu, Octavia Camps, David Bau, Pu Zhao, Yanzhi Wang

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 50

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/20/2026, 4:31:39 AM

Summary

This study investigates the mechanistic interpretability of LLaMA 3.1-8B's ability to perform structure-aware numerical reasoning. Using a custom sequence modeling task where models must predict numbers based on repeating first-difference patterns, the authors demonstrate that LLaMA 3.1-8B achieves high predictive accuracy. Probing experiments reveal that the model explicitly encodes first differences and final numbers in localized internal representations. Activation patching further shows that the model retrieves these first differences via an induction-like mechanism and adds them to the current value to extrapolate sequences, indicating a form of concept induction and arithmetic composition rather than mere memorization.

Entities (8)

LLaMA-3.1-8B · model · 100%Mechanistic Interpretability · methodology · 98%Activation Patching · method · 95%First Difference · concept · 95%Linear Probing · method · 95%Induction Circuit · mechanism · 92%NDIF · software · 90%NNsight · software · 90%

Relation Signals (6)

LLaMA-3.1-8B → analyzedby → Mechanistic Interpretability

confidence 98% · To study this, we investigate Llama 3.1-8B from a mechanistic interpretability point of view.

LLaMA-3.1-8B → computes → First Difference

confidence 95% · Probing reveals that the model computes and stores first differences in its internal representations without explicit supervision

LLaMA-3.1-8B → usesmechanism → Induction Circuit

confidence 92% · Activation patching reveals that Llama retrieves the relevant first-difference with a mechanism similar to an induction circuit

LLaMA-3.1-8B → performsarithmetic → First Difference

confidence 90% · subsequently adds it to the current value... performs the final arithmetic step by adding that offset to the last number

LLaMA-3.1-8B → usestool → NNsight

confidence 90% · We extensively use NNsight and NDIF framework (6) for tracing activations

LLaMA-3.1-8B → usestool → NDIF

confidence 90% · We extensively use NNsight and NDIF framework (6) for tracing activations

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Recent work has shown that large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction. While LLMs display in-context learning capabilities, the mechanisms with which they accomplish time-series prediction remain unclear. Specifically, whether they truly understand the underlying structure, which at a minimum requires reasoning over first differences in the sequence of numbers. To study this, we investigate Llama 3.1-8B from a mechanistic interpretability point of view. Mechanistic interpretability is an emerging field concerned with the reverse engineering of the algorithms learned by neural networks such as LLMs. To assess Llamas' numerical sequence modeling capabilities and to facilitate our mechanistic interpretability analysis, we create a sequence modeling task that cannot be solved without picking up structural cues. Specifically, we sample n random numbers and repeat them with an offset. We find that Llama displays strong performance on our tasks suggesting that it can pick up on the underlying structure. To understand the mechanisms that allow it to do so, we perform probing experiments and activation patching based counterfactual analysis. Probing reveals that the model computes and stores first differences in its internal representations without explicit supervision, indicating that it tracks structural information about the sequence. Activation patching reveals that Llama retrieves the relevant first-difference with a mechanism similar to an induction circuit and subsequently adds it to the current value. Notably, our work represents one of the first studies to identify this form of concept induction in LLMs.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

46,624 characters extracted from source content.

Expand or collapse full text

Mechanistic Interpretability of Structure-Aware Numerical Reasoning in LLaMA 3.1–8BConference: The fourth International Workshop on Rich Media with Generative AI ; November 10–14, 2026; Rio de Janeiro, BrazilThe fourth International Workshop on Rich Media with Generative AI (RichMediaGAI ’26), November 10–14, 2026, Rio de Janeiro, BrazilDOI: 10.1145/3841458.3841545ISBN: 979-8-4007-2947-8/2026/11CCS: Computing methodologies Natural language processingCCS: Time SeriesCCS: Mechanistic Interpretability Rahul Chowdhury OrcID: 0009-0008-1575-6077 Affiliation: Northeastern University , Boston , MA , USA email: chowdhury.rah@northeastern.edu , Timothy A Rupprecht Affiliation: EmbodyX Inc. , San Mateo , CA , USA email: tarupprecht@gmail.com , Senhao Cao Affiliation: Northeastern University , Boston , MA , USA email: senhao.cao@gmail.com , Jiahao Liu Affiliation: Mobi.ai , Boston , MA , USA email: jiahao@takemobi.com , Octavia Camps Affiliation: Northeastern University , Boston , MA , USA email: O.Camps@northeastern.edu , David Bau Affiliation: Northeastern University , Boston , MA , USA email: davidbau@northeastern.edu , Pu Zhao Affiliation: Northeastern University , Boston , MA , USA email: p.zhao@northeastern.edu and Yanzhi Wang Affiliation: Northeastern University , Boston , MA , USA email: yanzhiwang@northeastern.edu 2026; © c Abstract. Recent work has shown that large language models (LLMs) exhibit strong numerical sequence modeling capabilities and show promise in time-series prediction. While LLMs display in-context learning capabilities, the mechanisms with which they accomplish time-series prediction remain unclear. Specifically, whether they truly understand the underlying structure, which at a minimum requires reasoning over first differences in the sequence of numbers. To study this, we investigate Llama 3.1-8B from a mechanistic interpretability point of view. Mechanistic interpretability is an emerging field concerned with the reverse engineering of the algorithms learned by neural networks such as LLMs. To assess Llamas’ numerical sequence modeling capabilities and to facilitate our mechanistic interpretability analysis, we create a sequence modeling task that cannot be solved without picking up structural cues. Specifically, we sample n random numbers and repeat them with an offset. We find that Llama displays strong performance on our tasks suggesting that it can pick up on the underlying structure. To understand the mechanisms that allow it to do so, we perform probing experiments and activation patching based counterfactual analysis. Probing reveals that the model computes and stores first differences in its internal representations without explicit supervision, indicating that it tracks structural information about the sequence. Activation patching reveals that Llama retrieves the relevant first-difference with a mechanism similar to an induction circuit and subsequently adds it to the current value. Notably, our work represents one of the first studies to identify this form of concept induction in LLMs. †c-license: by 1. Introduction Figure 1. Illustrates how the model identifies a repeated first-difference (delta) pattern after differencing, then extrapolates by locating and copying the delta that immediately follows the final observed one, and adding it to last number. Large Language Models (LLMs) can zero-shot extrapolate sequences, and generate novel sequences and functions in response to complex queries (19; 25; 27; 45; 39; 2). How do they do this? Did they encounter these trends during training and simply extrapolate using a rule learned specifically for those patterns? Or did they learn to infer the underlying structure of sequence data more generally? If LLMs can understand the ordinal structure of numerical data — identifying a hidden pattern, reasoning about it, and applying a consistent extrapolation algorithm — the implications extend well beyond memorization. It would suggest these models are learning abstract structural reasoning, a capability with relevance far beyond language: reasoning over time-series data, understanding geometric relationships between image pixels, or maintaining structural coherence across connected points in 3D data (21; 23; 26; 22; 28; 24; 25; 37; 38). In this work, we study the LLaMA 3.1–8B model through the lens of mechanistic interpretability. To probe its structural reasoning capabilities, we introduce a non-trivial sequence modeling task, shown in Figure 2, that cannot be solved via token-level copying through induction heads (4). Our interpretability analysis combines linear probing with activation patching. Probing shows that first-difference representations are locally stored and linearly decodable across layers. Patching experiments further reveal that the model identifies structurally critical tokens, retrieves the correct offset through induction over these latent differences, and performs the final arithmetic step by adding that offset to the last number in the sequence. These computations are precise, localized, and structurally grounded. To our knowledge, this is the first mechanistic interpretability study to uncover an internal mechanism by which an LLM performs induction over latent structure in numerical sequences—where extrapolation arises from arithmetic operations over internal representations retrieved through, and we illustrate it in Figure 1. Figure 2. (a) Shows the input signal, which consists of all unique values which means no number appears twice in the sequence. (b) Shows the deltas of the input signal, revealing that a cycle or pattern emerges only after applying the differencing operation. 2. Related Work 19 showed that LLMs can perform sequence completion tasks, attributing this ability to in-context learning. However, they did not examine whether such performance stems from memorization, nor did they investigate the underlying mechanisms or the model’s understanding of temporal structure. (10) proposed LLM-Time that explored ways to make a pre-trained LLMs fit for time-series forecasting, demonstrating that pretrained LLMs can perform well on time-series forecasting tasks without fine-tuning. Time-GPT (7) is a foundation model trained exclusively on time-series data, and Lag-Llama (20) is another time-series-specific transformer. While these models highlight effectiveness of transformers for sequence modeling (44; 36; 34; 35; 15; 14; 16; 43; 17; 11; 33), they fall outside our study scope, as we focus on general-purpose LLM. 1 and 12 examined in-context learning using toy models but did not explore whether LLMs can infer sequential structure in tasks they were not explicitly trained for. (13) applied mechanistic interpretability to identify shared circuits in simple sequence continuation tasks, such as extending short increasing number sequences. While (13) demonstrated LLMs’ ability to handle familiar patterns, it did not address whether these models can reason over more abstract or irregular numerical structures. In contrast, our work investigates whether a LLM can recognize and extrapolate more complex sequential structures—such as sequences defined by arbitrary first differences. This goes beyond surface-level extension and explores the model’s ability to infer and extend abstract structure in number sequences. 3. Problem Setting 3.1. Dataset We design a dataset to have strings containing numbers followed by a comma, and each instance finishes with a comma so that the predicted next token is a number. The numbers in the whole dataset were between 0 and 999. We construct each sequence with first differences from integers in the set containing all integers between −9-9 and 99 inclusive, excluding 00. Each sequence consists of unique numbers whose deltas are also unique up to the 17th position, after which the delta repeats. We design the dataset to specifically test whether the LLM can uncover latent numerical structure and specifically track delta within a sequence. As we illustrate in Figure 2 and in an abbreviated form in Table 1, each sequence comprises two segments: the first is a random walk with unique deltas and no apparent pattern; the second segment reuses the same set of deltas, in the same order, as the first segment. This structure ensures that the model must recognize the pattern in delta and identify the unique location from which the model can copy the delta from. Index 0 1 2 3 4 5 23 24 25 26 27 31 32 33 34 35 36 37 55 56 57 58 59 Token S 403 , 405 , 400 411 , 417 , 409 416 , 436 , 438 , 433 444 , 450 , 442 Delta 2 -5 7 6 -8 -2 20 2 -5 7 6 -8 Table 1. Table shows selected index positions, tokens, and their corresponding first differences from the tokenized input. A recurrence in the first difference after token 33 marks the onset of a repeating cycle. Algorithm 1 Predict Next Number via First-Difference Pattern Detection Input: Token sequence S=[BOS,s1,s2,…,sn,]S=[BOS,s_1,s_2,…,s_n ,] Output: Predicted next number sn+1s_n+1 1: Compute first differences: 2: D=[s2−s1,s3−s2,…,sn−sn−1]D=[s_2-s_1,\ s_3-s_2,\ …,\ s_n-s_n-1] 3: Identify repeating pattern P in D 4: Locate the index k where P first appears in D 5: Determine the phase position p of sns_n within pattern P 6: Let dp=D⁡[p]d_p=D[p] be the corresponding first difference 7: Compute prediction: sn+1=sn+dps_n+1=s_n+d_p 8: return sn+1s_n+1 3.2. Model We conduct all experiments using LLaMA 3.1–8B (9), a transformer (30) based model with 32 layers and 32 attention heads per layer. One key reason for choosing this model is that its tokenizer represents each integer from 0 to 999 as a single token. This property makes the analysis more tractable by reducing the number of tokens per sequence, thereby simplifying both intervention and observation during interpretability experiments. As a result, LLaMA 3.1–8B was used consistently across all experiments. 3.3. Software We extensively use NNsight and NDIF framework (6) for tracing activations, applying interventions, and analyzing internal representations of the LLaMA 3.1–8B model. 3.4. Performance Evaluation We evaluate LLaMA 3.1-8B on the sequence prediction task as shown in Figure 1 and Figure 2, and we formally describe it in Algorithm 1. We prompt the model to predict the 30th number given the first 29 elements of each input sequence. The evaluation dataset consists of 10,000 sequences, each formatted as a comma-separated string of integers. Every instance ends with a comma, indicating that the next token to be predicted should be a number. Due to this formatting, we tokenize each number and each comma separately, as we illustrate in abbreviated form in Table 2. Including the initial special token, the model processes 59 tokens before generating the 60th token as output. This prediction corresponds to the 30th number in the sequence. 3.4.1. Metrics We employ two complementary evaluation metrics: Mean Absolute Error (MAE) quantifies average prediction error: (1) MAE=1n∑i=1n|yi−y^i|,MAE= 1n _i=1^n |y_i- y_i |, where yiy_i and y^i y_i denote the ground truth and predicted values, respectively, and n is the number of instances. Coefficient of Determination (R2R^2) measures explained variance: (2) R2=1−∑i=1n(yi−y^i)2∑i=1n(yi−y¯)2,R^2=1- _i=1^n(y_i- y_i)^2 _i=1^n(y_i- y)^2, where y¯ y is the mean of ground truth. Values close to 1 indicate strong predictive performance, while values near 0 or negative suggest performance close to or worse than a naive mean predictor. 3.4.2. Results LLaMA 3.1-8B achieves an MAE of 4.2748 and an R2R^2 of 0.9958 on our evaluation dataset. The high R2R^2 value indicates that the model explains approximately 99.58% of the variance in the target sequences, demonstrating strong pattern recognition capabilities for numerical sequence prediction. 4. Is Numerical Information Represented Locally in the Model? Figure 3. Layerwise linear decodability of first differences and final numbers. Mean absolute error (MAE; top) and coefficient of determination (R2R^2; bottom) for linear probes trained to decode delta (left) at 27th position and final number (right) at 57th position from hidden states across layers. Solid green lines denote performance on true targets; dashed red lines denote control probes trained on shuffled labels. Delta show consistently strong linear decodability, while final number representations gradually lose decodability. To predict the next number, the model must add the final observed value to a previously seen first difference. This raises a core interpretability question: does the model explicitly encode these numerical components in structured, localized representations, or does it rely solely on implicit pattern recognition? We use probing experiments to assess the representational content of the model’s hidden states. Unlike patching—which tests whether an activation is functionally necessary by measuring how altering it affects the model’s output—probing asks whether specific information is present in the representation, regardless of whether the model ultimately uses it. Here, we probe for the presence and location of numerically meaningful quantities in the hidden layers by training linear regressors to decode two key values: (1) the first difference from the 27th token position (where it is expected to be stored), and (2) the final number from the 57th token position. If successful, this would indicate that the model explicitly encodes these arithmetic components in localized, interpretable forms, even without task-specific supervision, and uses these interpretable representations for prediction. 4.1. Probing Setup We collect hidden states from two key token positions: the 27th token position, which is expected to encode the first difference that must be added to the final number to predict the next number, and the 57th token position, which contains the final number in the input sequence. At each layer, we train linear regression models using ordinary least squares on 4,500 training samples and 500 held-out samples for testing: (3) ^=argmin⁡‖−‖2, β= *arg\,min_ β \|Y-X β \|^2, where ∈Rn×dX ^n× d is the matrix of hidden states (with d=4096d=4096 dimensional representations across n=4500n=4500 training examples), ∈RnY ^n is the target vector (either the first difference or the final number), and ∈Rd β ^d is the learned weight vector. To establish baselines, we implement control conditions that test whether observed decodability reflects genuine localization or spurious correlations. For first difference decoding, we train control probes using activations from the 57th token to predict the first difference of 27th position, testing whether first difference information is accessible from positions nearer to the last token position. For final number decoding, we train control probes using activations from the 27th token to predict the number at the 57th position. These ensure that successful decoding reflects position-specific encoding rather than global numerical information distribution. 4.2. Results Figure 3 presents the results of probing experiments. The top row reports the MAE of the linear regression model on held-out test samples, while the bottom row shows the R2R^2. We compute these metrics at each layer to evaluate the linearity of the target quantities throughout the model’s depth. The results confirm that both the first difference (at the 27th token position) and the final number (at the 57th token position) are linearly decodable from the hidden states, as evidenced by low MAE values and high R2R^2 scores approaching 1. Notably, the first difference becomes linearly accessible starting after the 7th layer, where the MAE drops sharply and approaches its minimum, while the R2R^2 score increases and stabilizes above 0.8. Although performance slightly decays with depth, it remains relatively stable across the first 16 layers. In contrast, the linear decodability of the final number at the 57th position shows a gradual decline across deeper layers, with MAE increasing and R2R^2 decreasing after layer 8. This suggests that the final number representation becomes increasingly entangled with other contextual information as it propagates through model. Crucially, the control conditions demonstrate substantially worse performance across all layers for both targets. For first differences, control probes using 57th token activations achieve negative R2R^2 values and high MAE, indicating no meaningful linear relationship. Similarly, control probes attempting to decode final numbers from 27th token activations show consistently poor performance. This confirms that the observed decodability is position-specific and not due to general numerical information in the representations. Together, these results indicate that the numerical components necessary for the model’s arithmetic composition-the first difference and the final number-are linearly encoded in specific locations. 5. Is the Next Number Computed by Adding a Retrieved First Difference? Figure 4. Illustrates the query-hidden state patching technique that we use to test whether the model performs arithmetic composition by retrieving a first difference from an earlier position. The clean sequence (bottom) contains a structured delta pattern, while the corrupt sequence (top) is flat-valued. We transplant activations from the clean run into the corrupt run from a given layer l onward. We patch hidden states (shown with blue arrows) at layers l to L at one token position at a time except the beginning-of-sequence and final two tokens. We patch the query vector of the final token (shown with orange arrows) from the clean run at layers l+1l+1 to L, ensuring the model attends to the correct location encoding the target first difference. This setup isolates and tests the causal contribution of induction and addition behaviors in the model’s prediction pipeline. Having established that the model encodes first differences and final numbers in localized representations, we now ask whether it functionally uses those representations to perform arithmetic composition. We test whether the model retrieves a previously seen first difference from a particular token position and perform arithmetic operation on it. To answer this question, we use activation patching (32; 18; 31; 42; 40; 41; 8; 5), following the patching methodology of (3). We transplant activations from a clean sequence into a corrupt one to determine if the model retrieves the delta from the hypothesized position (delta that follows the latest one) and adds that to a new number of an unseen sequence. Clean vs. Corrupt Sequence We define two types of input sequences: • Clean sequence: xclean=(x1,…,x29)x^clean=(x_1,…,x_29), where the deltas Δi _i begin to repeat after i=17i=17, and the model predicts:x30=x29+Δ∗x_30=x_29+ ^* where Δ∗ ^* is the difference retrieved from an earlier position. In tokenized form, x29x_29 appears at token 57, and the output x30x_30 is generated at token 58. • Corrupt sequence: xcorrupt=(100,100,…,100)x^corrupt=(100,100,…,100), so that Δi=0 _i=0 for all i. What makes the corrupt sequence particularly well-suited for evaluating the extraction and addition of the first difference is its deliberately flattened structure, which eliminates any intrinsic gradient or variation in the input. The corrupt sequence serves as an ideal control, as it lacks the algorithm present in the clean sequence, allowing us to test whether the model can still retrieve and apply a meaningful first difference from an alternate source. 5.1. Patching Setup Let Ht(l)H_t^(l) and Qt(l)Q_t^(l) denote the hidden state and query vector at token t, layer l. We patch hidden states to all tokens except the beginning token, the final number token (57), and the final token (58): Htcorrupt,(l:L)←Htclean,(l:L)H_t^corrupt,(l:L)← H_t^clean,(l:L) The query vector of final token (58) is patched from one layer above: Q58corrupt,(l+1:L)←Q58clean,(l+1:L)Q_58^corrupt,(l+1:L)← Q_58^clean,(l+1:L) In particular, following (3), we patch layer l and all subsequent layers, as we illustrate in Figure 4. Patching downstream layers ensures that transient causal signals are preserved and not suppressed as noise by later layers, thereby, improving sensitivity to the influence of critical nodes on the final prediction. For the final token, we specifically patch the query vector across layers l+1l+1 through L. This is necessary because the corrupt sequence does not learn in-context to retrieve target first differences from earlier tokens. By patching the query vector of the final token from the clean forward pass, we ensure that the model attends to the position encoding the appropriate first difference, thereby, facilitating the intended retrieval behavior. 5.2. Causal Metric To quantify causal influence, we use difference in predicted probabilities of counterfactual label ycf=100+Δ∗y_cf=100+ ^* as evaluation metric: (4) ΔP=Ex∼[Ppatched(ycf∣x)−Pcorrupt(ycf∣x)] P=E_x [P_patched(y_cf x)-P_corrupt(y_cf x) ] where Ppatched(ycf∣x)P_patched(y_cf x) is the probability assigned to the counterfactual label ycfy_cf under the patched forward pass, and Pcorrupt(ycf∣x)P_corrupt(y_cf x) is the corresponding probability under the corrupt forward pass. A higher ΔP P indicates stronger causal contribution of the patched components to the model’s ability to predict the correct label. 5.3. Results Figure 5 presents the results of the patching experiment conducted on 100 data instances with zero MAE to ensure a strong response signal. We observe a significant increase in the probability difference score (ΔP P), highlighted in red in Figure 5, at the 27th token position, beginning from layer 14 onward. This indicates that, from layer 15 onward, the query vector of the final token (position 58) causally interacts with the hidden state at position 27. The arithmetic operation is highly localized: only at these layers does the model cleanly separate the corrupt final number and compose it with the correct first difference. Earlier interventions disrupt this alignment due to representational entanglement between the corrupt and patched sequences, which impairs computation. This finding supports the hypothesis that the model performs induction over first differences by locating and copying a locally stored delta representation—analogous to the [A][B]…[A][?] pattern discussed in (4)—that follows the final observed one, though this behavior operates at a structural rather than token level. The arithmetic operation that follows induction—adding the copied delta to the final number at position 57 to generate the next token—is generalizable, as the model performs it even in the absence of a guiding algorithm in the corrupt sequence. This confirms that the model engages in both induction and arithmetic composition in numerical sequence extrapolation. Figure 5. Heatmap of probability difference scores (ΔP P) across layers and token positions, following patching of the final query vector from layers l+1l+1 to L and hidden states from layers l to L. A sharp causal effect emerges at token 27, highlighted in red, indicating that the final query attends to position 27 to retrieve the locally stored first difference that follows the last observed one. 6. Can the Model Identify Functionally Critical Tokens for Delta Retrieval and Extrapolation? Although previous experiments demonstrate that the model stores first differences in localized representations and retrieves the appropriate ones for arithmetic composition, they do not tell if the model uncovers the underlying algorithm. In particular, uncovering requires identifying the onset of a repeating pattern, aligning it with the final observed difference, and selecting the correct delta to apply. To investigate this, we use patching to isolate the causal contribution of each attention head and assess which heads drive successful extrapolation by routing value from appropriate positions. Patching Setup We use the same clean and corrupt sequences described in the previous section, where the clean input contains a repeating first-difference pattern and the corrupt input is a flat sequence of 100s. The target counterfactual label remains: ycf=100+Δ∗y_cf=100+ ^*. Let Qs(l,h)Q_s^(l,h), Kt(l,h)K_t^(l,h), Vt(l,h)V_t^(l,h), and WO(l,h)W_O^(l,h) denote the query, key, value vectors, and output projection matrix for head (l,h)(l,h) in layer l at token s attending to token positions t. The attention weights at head (l,h)(l,h) are computed as: αt,s(l,h)=softmax⁡((Qs(l,h))⊤Kt(l,h)dh) _t,s^(l,h)=softmax ( (Q_s^(l,h)) K_t^(l,h) d_h ) The corresponding output of attention head (l,h)(l,h) at token t is: zs(l,h)=WO(l,h)(∑t=1Tαt,s(l,h)⋅Vt(l,h))z_s^(l,h)=W_O^(l,h) ( _t=1^T _t,s^(l,h)· V_t^(l,h) ) To evaluate the causal effect of head (l,h)(l,h) at the final token position s=58s=58, we patch just its output, as we illustrate in Figure 9 in the appendix , from the clean run into the corrupt run: z58corrupt,(l,h)←z58clean,(l,h)z_58^corrupt,(l,h)← z_58^clean,(l,h) This setup isolates the contribution of a single attention head by substituting its output at the final token while preserving the rest of the corrupt context. 6.1. Causal Metric To determine which tokens most significantly contribute to the model’s prediction, we construct a profile of attention heads that integrates two key signals: (1) the causal effect of each head, and (2) the value-weighted attention mass it allocates across token positions. We compute the causal effect as the change in the model’s predicted probability for the counterfactual label when we patch head’s output at position s=58s=58 from the clean to the corrupt run, as we describe in the previous section. The second component—the value-weighted attention— identifies how much of its value is added to the residual stream through this head. We define the token importance score TItTI_t as: (5) TIt=1N∑n=1Nλn(l,h)⋅wn,t(l,h), _t= 1N _n=1^N _n^(l,h)· w_n,t^(l,h), where N is the number of evaluation samples, λn(l,h)=Ppatched(ycf∣xn)−Pcorrupt(ycf∣xn) _n^(l,h)=P_patched(y_cf x_n)-P_corrupt(y_cf x_n) is the causal effect of patching head (l,h)(l,h) in sample n, and wn,t(l,h)w_n,t^(l,h) is the value-weighted attention score at the attended position t. The value-weighted attention score wn,t(l,h)w_n,t^(l,h) is defined using the attention weights and value norms: (6) wn,t(l,h)=αt,s(l,h)⋅‖Vt(l,h)‖2,w_n,t^(l,h)= _t,s^(l,h)· \|V_t^(l,h) \|_2, This metric TItTI_t weights causal influence with value-aware attention, providing a principled measure of token-level functional relevance to the model’s output. 6.2. Results Figure 6 presents the results of the causal attention-based analysis over 100 sequences with zero MAE, and the top five most relevant tokens are identified as positions 57, 33, 27, 55, and 25. As we show in Table 1, these correspond to the critical points that the model needs for identification and extrapolation of the pattern: the final number (57), the onset of the repeating delta pattern (33), the delta that gets added to the final value (27), the number that the model needs to compute the last delta (55), and a token (25) that both matches the final delta and enables retrieval of the next one. These results also suggest the model emphasizes tokens that support first-difference alignment, with token 27 playing a key role in extrapolation. The pattern of attention supports the hypothesis that the model recognizes the pattern and must have first used the final first difference as a retrieval anchor to locate the appropriate delta, and, upon recognizing the repeating structure, may further leverage phase alignment and positional encoding to guide delta selection. Figure 6. Token importance scores (TItTI_t) across sequence positions. Peaks highlight the model’s use of structural alignment for pattern detection and delta retrieval. 7. Does the Model Prioritize Delta Induction Over Positional Cues? Prior experiments demonstrate that the model stores and composes first differences to extrapolate future values. However, they do not reveal whether retrieval is governed by semantic identity of delta or by its positional and phase-based alignment in input. To investigate this, we perform a key swapping intervention between the 27th and 25th tokens—two positions with high and comparable importance scores as shown in Figure 6. 7.1. Rotary Position Embedding (RoPE) Let x∈dx ^d denote an input vector, where d is even. Following the formulation introduced by (Su et al. 2024)(29), we partition x into d/2d/2 adjacent 2D pairs: x=[(x1(1),x1(2)),(x2(1),x2(2)),…,(xd/2(1),xd/2(2))]x= [\,(x_1^(1),x_1^(2)),\,(x_2^(1),x_2^(2)),\,…,\,(x_d/2^(1),x_d/2^(2))\, ] Let Wq,Wk∈d×dW_q,W_k ^d× d be the query and key projection matrices. For position m, the RoPE-transformed query is: Qm=RoPE(Wqx,m)=⨁i=1d/2R⁡(mθi)(Wqx)iQ_m=RoPE(W_qx,m)= _i=1^d/2R(m _i)\,(W_qx)_i and similarly, for position n, the RoPE-transformed key is: Kn=RoPE(Wkx,n)=⨁i=1d/2R⁡(nθi)(Wkx)iK_n=RoPE(W_kx,n)= _i=1^d/2R(n _i)\,(W_kx)_i Here, R⁡(mθi)R(m _i) is a 2D rotation matrix applied to the i-th 2D component: R⁡(mθi)=[cos⁡(mθi)−sin⁡(mθi)sin⁡(mθi)cos⁡(mθi)]R(m _i)= bmatrix (m _i)&- (m _i)\\ (m _i)& (m _i) bmatrix Here, R⁡(nθi)R(n _i) is a 2D rotation matrix applied to the i-th 2D component: R⁡(nθi)=[cos⁡(nθi)−sin⁡(nθi)sin⁡(nθi)cos⁡(nθi)]R(n _i)= bmatrix (n _i)&- (n _i)\\ (n _i)& (n _i) bmatrix The operator ⨁ denotes vector concatenation across the d/2d/2 rotated 2D components: ⨁i=1d/2vi=[v1;v2;…;vd/2] _i=1^d/2v_i=[v_1;\,v_2;\,…;\,v_d/2] where each vi∈2v_i ^2, yielding a final vector in dR^d. The query and key vectors are independently rotated using their respective absolute positions m and n, and the relative positional information gets encoded through their dot product Qm⊤KnQ_m K_n. 7.2. Patching Setup LLaMA 3.1-8B uses rotary position embedding (RoPE) (29) where positional information is local to key and query. We swap Keys K27clean,(l:L)↔K25clean,(l:L)K_27^clean,(l:L) K_25^clean,(l:L) in layer l and all subsequent layers to redirect positional and phase-based cues while preserving V27clean,(l:L)V_27^clean,(l:L) and V25clean,(l:L)V_25^clean,(l:L) by patching the values from the original run to maintain their first difference content across layers l+1l+1 through L, as we illustrate in Figure 10 in the appendix. This setup attempts to disentangle two types of cues bound to the delta representation. 7.3. Causal Metric To quantify the causal impact of an intervention, we measure the change in the model’s confidence for both the counterfactual label ycf=x29+Δ25y_cf=x_29+ _25 and the ground truth label ygt=x29+Δ27y_gt=x_29+ _27. The causal effect on the counterfactual label is defined as: (7) ΔPcf=Ex∼[Ppatched(ycf∣x)−Pcorrupt(ycf∣x)], P_cf=E_x [P_patched(y_cf x)-P_corrupt(y_cf x) ], and for the ground truth label: (8) ΔPgt=Ex∼[Ppatched(ygt∣x)−Pcorrupt(ygt∣x)]. P_gt=E_x [P_patched(y_gt x)-P_corrupt(y_gt x) ]. We also report the absolute probabilities assigned after the intervention: Ppatched(ycf∣x)P_patched(y_cf x) and Ppatched(ygt∣x)P_patched(y_gt x). These metrics together capture both the direction and magnitude of the model’s response to the intervention. 7.4. Results Figure 7 presents the probability difference following the key-swapping intervention K27clean,(l:L)↔K25clean,(l:L)K_27^clean,(l:L) K_25^clean,(l:L) that we conduct over 100 instances with zero MAE. The intervention leads to a significant increase in ΔPcf P_cf and a corresponding drop in ΔPgt P_gt, suggesting that phase or position-based cues contribute meaningfully for delta retrieval and composition. However, as shown in Figure 8, which reports the absolute probabilities after intervention, the drop in Ppatched(ygt)P_patched(y_gt) was not sufficient for Ppatched(ycf)P_patched(y_cf) to overtake it. This indicates that although key-based redirection influences the model’s behavior, it is ultimately not strong enough to override the original delta-based composition since the value vector must contain information that this first difference comes after the last observed one. The model’s affinity for delta-based composition supports the hypothesis that it first performed induction over first differences to learn the underlying structure: internally simulated the composition process, and identified the correct algorithm, potentially refining it through phase-based alignment. Figure 7. Shows key-swap intervention shifts confidence from ground truth to counterfactual, revealing the role of position-aware attention alignment in delta retrieval. Figure 8. Shows ground truth remains more probable post key-swap, indicating the model prioritizes delta-based composition over redirected attention. 8. Conclusion This work demonstrates the LLM, despite not being explicitly trained for high-precision numerical tasks, exhibit an emergent ability to perform structural extrapolation in numerical sequences without any explicit supervision. We show that the LLM is capable of recognizing trends, inferring position-dependent rules encoded as distinct first differences, and systematically composing these rules to generate accurate predictions over evolving input sequences. Our analyses reveal that the LLM exhibit reasoning behaviors, operating over latent structure. Patching provided strong evidence that the model identifies numerical patterns, computes first differences even before recurring structures emerge, stores these differences locally, and later retrieves them to generate accurate predictions. Probing confirmed that the observed effects are not incidental but reflect functional computations over structured internal representations. We believe these findings would invite future work on extending these insights to other reasoning tasks where latent structure plays a critical role. References Akyürek et al. (2022) E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661. Cited by: §2. Bubeck et al. (2023) S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: early experiments with gpt-4. arXiv preprint arXiv:2303.12712. Cited by: §1. Dumas et al. (2024) C. Dumas, V. Veselovsky, G. Monea, R. West, and C. Wendler How do llamas process multilingual text? a latent exploration through activation patching. In ICML 2024 Workshop on Mechanistic Interpretability, Cited by: §5.1, §5. Elhage et al. (2021) N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah A mathematical framework for transformer circuits. Transformer Circuits Thread. Note: https://transformer-circuits.pub/2021/framework/index.html Cited by: §1, §5.3. Finlayson et al. (2021) M. Finlayson, A. Mueller, S. Gehrmann, S. Shieber, T. Linzen, and Y. Belinkov Causal analysis of syntactic agreement mechanisms in neural language models. arXiv preprint arXiv:2106.06087. Cited by: §5. Fiotto-Kaufman et al. (2024) J. Fiotto-Kaufman, A. R. Loftus, E. Todd, J. Brinkmann, C. Juang, K. Pal, C. Rager, A. Mueller, S. Marks, A. S. Sharma, F. Lucchetti, M. Ripa, A. Belfki, N. Prakash, S. Multani, C. Brodley, A. Guha, J. Bell, B. Wallace, and D. Bau NNsight and ndif: democratizing access to foundation model internals. External Links: 2407.14561, Link Cited by: §3.3. Garza and Mergenthaler-Canseco (2023) A. Garza and M. Mergenthaler-Canseco TimeGPT-1. arXiv preprint arXiv:2310.03589. Cited by: §2. Goldowsky-Dill et al. (2023) N. Goldowsky-Dill, C. MacLeod, L. Sato, and A. Arora Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969. Cited by: §5. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, et al. The llama 3 herd of models. External Links: 2407.21783, Link Cited by: §3.2. Gruver et al. (2024) N. Gruver, M. Finzi, S. Qiu, and A. G. Wilson Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems 36. Cited by: §2. Guan et al. (2021) H. Guan, S. Liu, X. Ma, W. Niu, B. Ren, X. Shen, Y. Wang, and P. Zhao CoCoPIE: enabling real-time ai on off-the-shelf mobile devices via compression-compilation co-design. Communications of the ACM 64 (6), p. 62–68. Cited by: §2. Kantamneni et al. (2024) S. Kantamneni, Z. Liu, and M. Tegmark How do transformers" do" physics? investigating the simple harmonic oscillator. arXiv preprint arXiv:2405.17209. Cited by: §2. Lan et al. (2023) M. Lan, P. Torr, and F. Barez Towards interpretable sequence continuation: analyzing shared circuits in large language models. arXiv preprint arXiv:2311.04131. Cited by: §2. Li et al. (2023) Y. Li, C. Yang, P. Zhao, et al. Towards real-time segmentation on the edge. AAAI’23/IAAI’23/EAAI’23. External Links: ISBN 978-1-57735-880-0, Link, Document Cited by: §2. Li et al. (2022) Y. Li, P. Zhao, G. Yuan, X. Lin, Y. Wang, and X. Chen Pruning-as-search: efficient neural architecture search via channel pruning and structural reparameterization. arXiv preprint arXiv:2206.01198. Cited by: §2. Lin et al. (2025) J. Lin, A. Taherin, A. Akbari, A. Akbari, et al. Vote: vision-language-action optimization with trajectory ensemble voting. arXiv preprint arXiv:2507.05116. Cited by: §2. Liu et al. (2025) J. Liu, Z. Kong, P. Dong, C. Yang, et al. Structured agent distillation for large language model. arXiv preprint arXiv:2505.13820. Cited by: §2. Meng et al. (2022) K. Meng, D. Bau, A. Andonian, and Y. Belinkov Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems 35, p. 17359–17372. Cited by: §5. Mirchandani et al. (2023) S. Mirchandani, F. Xia, P. Florence, B. Ichter, D. Driess, M. G. Arenas, K. Rao, D. Sadigh, and A. Zeng Large language models as general pattern machines. arXiv preprint arXiv:2307.04721. Cited by: §1, §2. Rasul et al. (2023) K. Rasul, A. Ashok, A. R. Williams, A. Khorasani, G. Adamopoulos, R. Bhagwatkar, M. Biloš, H. Ghonia, N. Hassen, A. Schneider, et al. Lag-llama: towards foundation models for time series forecasting. In R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, Cited by: §2. Shen et al. (2025a) X. Shen, C. Han, Y. Zhou, et al. DraftAttention: fast video diffusion via low-resolution attention guidance. arXiv preprint arXiv:2505.14708. Cited by: §1. Shen et al. (2025b) X. Shen, W. Ma, J. Liu, et al. QuartDepth: post-training quantization for real-time depth estimation on the edge. In CVPR, Cited by: §1. Shen et al. (2026) X. Shen, W. Ma, Y. Zhou, et al. Fastcar: cache attentive replay for fast auto-regressive video generation on the edge. In ICLR, Cited by: §1. Shen et al. (2025c) X. Shen, Z. Song, Y. Zhou, et al. Lazydit: lazy learning for the acceleration of diffusion transformers. In AAAI, Cited by: §1. Shen et al. (2025d) X. Shen, Z. Song, Y. Zhou, et al. Numerical pruning for efficient autoregressive models. In AAAI, Cited by: §1, §1. Shen et al. (2025e) X. Shen Y. Wang et al. Efficient reasoning with hidden thinking. arXiv preprint arXiv:2501.19201. Cited by: §1. Shen et al. (2024) X. Shen, P. Zhao, Y. Gong, Z. Kong, Z. Zhan, Y. Wu, M. Lin, C. Wu, X. Lin, and Y. Wang Search for efficient large language models. In NeurIPS, Cited by: §1. Shen et al. (2025f) X. Shen, H. Zheng, Y. Gong, et al. Sparse learning for state space models on mobile. In ICLR, External Links: Link Cited by: §1. Su et al. (2024) J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu Roformer: enhanced transformer with rotary position embedding. Neurocomputing 568, p. 127063. Cited by: §7.1, §7.2. Vaswani (2017) A. Vaswani Attention is all you need. Advances in Neural Information Processing Systems. Cited by: §3.2. Vig et al. (2020) J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. Shieber Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems 33, p. 12388–12401. Cited by: §5. Wang et al. (2023) K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §5. Wang et al. (2018) S. Wang, X. Wang, S. Ye, P. Zhao, and X. Lin Defending dnn adversarial attacks with pruning and logits augmentation. In 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP), p. 1144–1148. Cited by: §2. Wu et al. (2022) Y. Wu, Y. Gong, P. Zhao, et al. Compiler-aware neural architecture search for on-mobile real-time super-resolution. In ECCV, p. 92–111. Cited by: §2. Yang et al. (2023) C. Yang, P. Zhao, Y. Li, et al. Pruning parameterization with bi-level optimization for efficient semantic segmentation on the edge. In CVPR, Cited by: §2. Zhan et al. (2021) Z. Zhan, Y. Gong, P. Zhao, G. Yuan, et al. Achieving on-mobile real-time super-resolution with neural architecture and pruning search. In ICCV, p. 4821–4831. Cited by: §2. Zhan et al. (2024a) Z. Zhan, Z. Kong, Y. Gong, et al. Exploring token pruning in vision state space models. In NeurIPS, External Links: Link Cited by: §1. Zhan et al. (2024b) Z. Zhan, Y. Wu, Y. Gong, et al. Fast and memory-efficient video diffusion using streamlined inference. In NeurIPS, External Links: Link Cited by: §1. Zhan et al. (2024c) Z. Zhan, Y. Wu, Z. Kong, et al. Rethinking token reduction for state space models. In EMNLP, Miami, Florida, USA, p. 1686–1697. External Links: Link Cited by: §1. Zhao et al. (2026a) L. Zhao, X. Jiang, X. Xiao, Q. Fan, L. Lu, Y. Wang, X. Lin, O. Camps, P. Zhao, and J. Gu Hieramp: coarse-to-fine autoregressive amplification for generative dataset distillation. arXiv preprint arXiv:2603.06932. Cited by: §5. Zhao et al. (2026b) L. Zhao, Y. Wu, Y. Gong, Y. Wang, and P. Zhao OmniMem: scalable and adaptive memory retrieval for long video generation. arXiv preprint arXiv:2605.30519. Cited by: §5. Zhao et al. (2025a) L. Zhao, Y. Wu, X. Jiang, J. Gu, Y. Wang, X. Xu, P. Zhao, and X. Lin Taming diffusion for dataset distillation with high representativeness. arXiv preprint arXiv:2505.18399. Cited by: §5. Zhao et al. (2025b) P. Zhao, D. Gunawan, X. Shen, Z. Zhan, et al. Efficient and accurate post-training sparsification of large language models with proximal operators. In Proceedings of the 3rd International Workshop on Rich Media With Generative AI, RichMediaGAI ’25, p. 11–19. External Links: ISBN 9798400720444 Cited by: §2. Zhao et al. (2024a) P. Zhao, X. Shen, Z. Kong, Y. Shen, S. Chang, T. Rupprecht, L. Lu, E. Nan, C. Yang, Y. He, et al. Fully open source moxin-7b technical report. arXiv preprint arXiv:2412.06845. Cited by: §2. Zhao et al. (2024b) P. Zhao, F. Sun, X. Shen, et al. Pruning foundation models for high accuracy without retraining. In Findings of EMNLP 2024, External Links: Link, Document Cited by: §1. Appendix Appendix A Illustrating Patching Experiments Figure 9. Illustrates the head-level patching intervention that we use to isolate the functional role of individual attention heads in pattern recognition and delta retrieval. The clean sequence (bottom) contains structured first-difference patterns, while the corrupt sequence (top) is flat-valued. At each layer l, we patch only the output of a single attention head (at the final token position) from the clean forward pass into the corrupt one (shown with orange arrow), while all other heads and components remain unchanged. This targeted intervention tests whether the selected head causally contribute to the recognition of repeating structure, retrieval of a critical first difference, and contributes to accurate extrapolation in the final prediction. Figure 10. Illustrates the key-swapping intervention that we use to disentangle the model’s reliance on phase (positional) cues versus delta-based representations during extrapolation. We swap keys from token positions 25 and 27—both identified as causally important—are swapped from layer l onward (indicated by orange and gray arrows), while preserving the respective value vectors from layer l onward. This setup tests whether the model selects the correct first difference for addition based on the original delta identity stored in the value vector or shifts its prediction in response to the redirected positional cue, thereby revealing its bias toward phase.

Instant research discovery