Paper deep dive
DMDIntel: Interpreting Large Language Models via Dynamic Mode Decomposition
Amogh Joshi, Animesh Mukherjee, Sergey Utyuzhnikov
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/14/2026, 5:51:36 AM
Summary
The paper introduces DMDIntel, a framework for interpreting Large Language Models (LLMs) by treating their hidden state evolution as a dynamical system. Using Dynamic Mode Decomposition (DMD) and Higher-Order DMD (HODMD), it decomposes hidden states into spatio-temporal modes to rank input tokens by their influence on classification outputs. Experiments across three model families and datasets show DMDIntel outperforms baselines like PCA, Integrated Gradients, and SHAP.
Entities (12)
Relation Signals (11)
DMDIntel → uses → Dynamic Mode Decomposition
confidence 95% · DMDIntel which uses dynamic mode decomposition (DMD) to make the predictions made by LLMs in a classification task interpretable.
DMDIntel → evaluatedon → Sentiment
confidence 90% · evaluating it across three distinct LLM families ... using three diverse text classification datasets ... Sentiment
DMDIntel → evaluatedon → FakeEdit
confidence 90% · evaluating it across three distinct LLM families ... using three diverse text classification datasets ... FakeEdit
DMDIntel → evaluatedon → HateXplain
confidence 90% · evaluating it across three distinct LLM families ... using three diverse text classification datasets ... HateXplain
DMDIntel → outperforms → SHAP
confidence 90% · ranked attribution of input tokens obtained using DMDIntel by far outperforms state-of-the-art techniques such as ... SHAP.
DMDIntel → outperforms → Integrated Gradients
confidence 90% · ranked attribution of input tokens obtained using DMDIntel by far outperforms state-of-the-art techniques such as ... integrated gradients
DMDIntel → outperforms → Principal Component Analysis
confidence 90% · ranked attribution of input tokens obtained using DMDIntel by far outperforms state-of-the-art techniques such as principal component analysis
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In this work, we introduce DMDIntel which uses dynamic mode decomposition (DMD) to make the predictions made by LLMs in a classification task interpretable. It develops an input attribution pipeline, that first decomposes the hidden states of an LLM into prominent patterns, also known as modes, and then associates ranks to the input tokens based on the projection values on those modes. Rigorous experiments across three datasets and three model families consistently show that the ranked attribution of input tokens obtained using DMDIntel by far outperforms state-of-the-art techniques such as principal component analysis, integrated gradients and SHAP.
Tags
Links
- Source: https://arxiv.org/abs/2608.13048v1
- Canonical: https://arxiv.org/abs/2608.13048v1
Trouble viewing inline? Open PDF directly →
Full Text
68,215 characters extracted from source content.
Expand or collapse full text
DMDINTEL: Interpreting Large Language Models via Dynamic Mode Decomposition Amogh Joshi IIT Kharagpur, India University of Manchester, UK Animesh Mukherjee IIT Kharagpur, India Sergey Utyuzhnikov University of Manchester, UK Abstract In this work, we introduce DMDINTEL which uses dynamic mode decomposition (DMD) to make the predictions made by LLMs in a clas- sification task interpretable. It develops an in- put attribution pipeline, that first decomposes the hidden states of an LLM into prominent patterns, also known as modes, and then asso- ciates ranks to the input tokens based on the projection values on those modes. Rigorous ex- periments across three datasets and three model families consistently show that the ranked at- tribution of input tokens obtained using DMD- INTEL by far outperforms state-of-the-art tech- niques such as principal component analysis, integrated gradients and SHAP. 1 Introduction Transformer language models (Vaswani et al., 2017) redefined the processing of natural language by introducing self-attention. Following this, many modifications have been made to the transformer ar- chitecture, such as encoder-only and decoder-only models (Devlin et al., 2019; Radford et al., 2018). Due to an increase in the complexity of these mod- els, interpretability remains a huge challenge. A significant body of literature has been dedicated to dissecting and interpreting the internal mecha- nisms of attention layers in transformer-based ar- chitectures. Traditional interpretability methods, such as Integrated Gradients (IG) (Sundararajan et al., 2017), SHAP (Lundberg and Lee, 2017), etc., are often used for input attribution in LLMs. Although these methods are effective in identify- ing “what” the model looks at, they treat the tokens as discrete entities and often overlook the sequen- tial evolution of the latent information, providing a fragmented snapshot rather than a continuous nar- rative of model’s reasoning. In addition to attribution, there are some attempts to decode the structural logic of the MLP component of the decoder layer. Geva et al. (2021) suggests that the MLP module acts as a key-value memory bank, where the first linear layer is a pattern de- tector (keys) and the second is a value provider that updates the residual stream. Hernandez et al. (2023) states that much of the black-box computa- tion within an MLP can be approximated via linear operations in high-dimensional space. However, even these structural insights frequently treat each token’s hidden state as an isolated vector, failing to capture the logic that builds across a sequence. To address these issues, in this paper, we attempt to take advantage of masked self-attention, which in- vokes the sequential nature of decoder-only LLMs by analyzing hidden states as they evolve token by token. By viewing these hidden states as tra- jectories, we can treat the supervised fine-tuned LLM as a dynamical system where the internal pro- cessing resembles a flow in the embedding space. We use Dynamic Mode Decomposition (DMD) (Schmid, 2010) to find a surrogate linear opera- tor,A, which approximates the complex model underlyingF. Specifically, we model the state transition,h t+1 =F (w ≤t ), using a linear relation- shiph t+1 = Ah t , wherew ≤t represents the tokens until the current token indextandh t represents the hidden state of the model at the token indext. This linear operator is then decomposed using DMD into interpretable spatio-temporal modes. These modes can be thought of as the low-dimensional structures that manifest when a model is supervised fine-tuned for a particular downstream task. We then use the dominant modes for input attribution to identify the most influential tokens. Thus, we can think of these tokens as the main drivers of the model’s input processing, as it prepares the output by reading the input tokens. Key contributions: Our key contributions in this work are as follows. • We introduce DMDINTEL, an input attri- bution framework for supervised fine-tuned arXiv:2608.13048v1 [cs.AI] 13 Aug 2026 LLMs that extracts the primary tokens driv- ing a given output classification. We show that the hidden states of a supervised fine- tuned model, obtained from thedown_proj of its multi-layer perceptron (MLP) compo- nent, show interpretable patterns when decom- posed into lower-dimensional spatio-temporal modes. The ranking of tokens by their projec- tions onto these DMD modes demonstrates that certain modes serve as robust indicators of the model’s information flow. •We validate the versatility and robustness of our approach by evaluating it across three distinct LLM families of varying parameter scales using three diverse text classification datasets. •To the best of our knowledge, this is the first work to analyze the sequential evolu- tion of hidden states in decoder-only LLMs as a dynamical system for input attribution. We demonstrate that DMDINTEL consistently identifies the highest proportion of ground- truth tokens while preserving their ranked importance, outperforming baseline methods such as PCA, IG, and SHAP. 2 Background Dynamic Mode Decomposition (DMD) (Schmid, 2010) was originally developed in fluid mechanics to extract coherent spatial structures from com- plex, non-linear flow fields. DMD is a purely data-driven, model-agnostic technique that requires no prior knowledge of the underlying system gov- erning equations. By operating directly on time- series observation snapshots, DMD constructs a best-fit linear operator that approximates the non- linear dynamics of the system. The resulting eigendecomposition yields a set of spatial DMD modes—representing the primary spatial structures along with complex eigenvalues that quantify their temporal growth, decay, and oscillation frequen- cies. A complete mathematical derivation and al- gorithmic setup of DMD is detailed in Appendix B. 3 Related work Interpretability of LLMs: Various analytical frameworks have been proposed to understand the inner workings of LLMs. Probing-based ap- proaches evaluate whether hidden representations capture structural or semantic features by train- ing linear classifiers on top of frozen model layers (Tenney et al., 2019). Beyond static probing, mech- anistic interpretability seeks to reverse-engineer specific network circuits, mapping exact weights and attention head paths to functional behaviors such as in-context pattern matching (Olsson et al., 2022) or factual retrieval (Meng et al., 2022). To observe how these internal features evolve into fi- nal predictions, methods such as the LogitLens decode intermediate hidden states directly into the vocabulary space. This concept is further refined in TunedLens (Belrose et al., 2023), which trains lin- ear adapters at each layer to more accurately map representation dynamics to the model’s predictive trajectory. DMD applications: Beyond fluid dynamics, DMD has been successfully applied to video pro- cessing scenarios. Kutz et al. (2017) established DMD’s capability for background modeling in video stream data, effectively separating the fore- ground from the background.Erichson et al. (2019) utilized a memory-efficient variant of DMD to achieve foreground-background separation in streaming video data. DMD for NLP tasks: In recent works, DMD has also been used for feature extraction to improve performance in classification tasks. For example, (Sachin Kumar et al., 2019) uses DMD to extract spatio-temporal features from text representations to capture the evolving sentiment trajectory across a sentence. Similarly, this dynamical perspective has been successfully applied to multimodal and spoken language tasks; for example, (Mao et al., 2020) utilizes DMD to derive spectral representa- tions of audio utterances. Vyshnav et al. (2020) evaluates DMD along with traditional sequence models to isolate consistent and underlying seman- tic patterns from highly noisy and user-generated social media text. Together, these works highlight the efficacy of DMD in capturing non-linear dy- namics across diverse linguistic modalities. 4 The DMDINTEL framework In this section, we outline the DMDINTEL frame- work which we propose for interpreting the predic- tions of decoder-only LLMs when they are used to perform various text classification tasks. We view the LLMs as a discrete-time dynamical sys- tem, where the evolution of hidden states acts as a first-order approximation of the LLMF, such that h k+1 ≈F (h k ). Subsequently, we use DMD (see Appendix B for more details) to approximateF, and finally decompose it to get a set of interpretable modes. 4.1 Formulation of the data matrix As an LLM processes a text sequence, it stores the information in the intermediate hidden states such thath t+1 = F (w ≤t ). The information can be considered as the combined representation of the instruction prompt plus the information from the actual input sentence in some non-linear fash- ion. This information is passed through all the layers, before passing through the unembedding matrix, followed by softmax to generate the actual token. We target a layer, and in that layer, we con- sider outputs from thedown_projof the multilayer perceptron. For an input sequence consisting of w 1 ,w 2 ,...,w n tokenized words, we collect the vec- tors from the selected layer to form the matrixX such that X = | || h 1 h 2 · h n | || (1) where each h i =F (w ≤i ). 4.2 Noise removal We find that the instruction fine-tuned models en- code a significant representation of the prompt tem- plate used during fine-tuning within the hidden rep- resentations of each token. As Hemati et al. (2017) proves the sensitivity of DMD to noise in the data, we denoise the data matrix obtained in Eq. 1 by re- moving the noise inflicted by the prompt template. For this, we pass an empty prompt consisting of the same exact system and user prompts without the input sentence, and then collect the hidden repre- sentation from the same layer from where we pick the token representations. We call this instruction bias,b. This bias is then subtracted from each h i to obtain the debiased estimate of the hidden representation of the actual token w i . ˆ h i = h i −b(2) Thus, our resulting data matrix is the collection of the debiased hidden states of each token. ˆ X = | || ˆ h 1 ˆ h 2 · ˆ h n | || (3) Algorithm 1 Heuristic layer selection Require:Fine-tuned LLMF; input sequencew = (w 1 ,w 2 ,...,w n ) ; instruction bias vectorb(obtained by passing an empty prompt throughF); cosine threshold τ = 0.25 Ensure: Selected layer index ℓ ∗ 1: LetL mid denote the middle-indexed decoder layer ofF, and L last the final decoder layer. 2: for each layer ℓ∈L mid , L mid + 1, ..., L last do 3:Extract thedown_projhidden representationh (ℓ) t for each token w t , without debiasing. 4:Compute the per-token cosine similarity with the bias vector: c (ℓ) t = ⟨h (ℓ) t ,b⟩ ∥h (ℓ) t ∥b∥ 5:Average over all tokens to obtain a layer-level simi- larity score: ̄c (ℓ) = 1 n n X t=1 c (ℓ) t 6: end for 7: Form the cosine-similarity curveC = ̄c (ℓ) L last ℓ=L mid . 8: Restrict the candidate set to layers whose similarity lies below the threshold: L τ = ℓ | ̄c (ℓ) < τ 9: WithinL τ , compute the discrete gradient ofC: ∇ ̄c (ℓ) = ̄c (ℓ+1) − ̄c (ℓ) 10:Identify the layer at the lower end of the steepest ascent inC: ℓ ∗ = arg max ℓ∈L τ ∇ ̄c (ℓ) 11: return ℓ ∗ 4.3 Layer selection Recall that a transformer has multiple layers, and each layer has its own representation of the input tokens. We define a heuristic to identify the de- coder layer whosedown_projMLP outputs best represent the actual input sentence, balancing input- token fidelity against instruction-following bias. The intuition behind this heuristic is as follows. Layers close to the middle layer of the transformer, L mid tend to be nearly orthogonal tob, indicat- ing that their representations are driven primarily by the actual input tokens. Asℓincreases toward L last (last transformer layer), the cosine similarity also rises, reflecting a shift in the model’s focus toward instruction-following rather than input con- tent. The selected layerℓ ∗ sits at the inflection point of this curveC– the point of highest local gradient still below the cosine similarity threshold τ = 0.25– where the model strikes the best bal- ance between encoding input-token semantics and attending to the instruction prompt. We verify that (a) Llama-3.2-3B-inst: FE(b) Llama-3.2-3B-inst: HX(c) Qwen3-4B-inst: FE (d) Qwen3-4B-inst: SNT(e) Qwen3-4B-inst: HX(f) Mistral-7B-v0.3-inst: SNT Figure 1: Cosine similarity curvesCacross different datasets and models.x-axis shows the layer indices starting from the middle layer of the model till the end.y-axis shows the average cosine similarity of the bias vector with the input token sentences. FE: FakeEdit, HX: HateXplain, SNT: Sentiment. this heuristic is consistent across all model families and datasets considered in this work (see example curvesCin Figure 1 for different datasets and mod- els discussed in Section 5). The procedure based on this heuristic is formalized in Algorithm 1, and the layers selected for different model and dataset settings are detailed in Table 1 Model FamilySentimentHateXplainFakeEdit Llama-3.2-3B-inst191818 Qwen3-4B-inst222323 Mistral-7B-v0.3-inst202722 Table 1: Selected optimal layer indices across different datasets and model families. 4.4 Input token attribution We use PyDMD, (Demo et al., 2018), a robust Python package, which consists of implementa- tions of various DMD algorithms, to obtain the approximate surrogateFof the LLM using the de- biased hidden state matrix. The modes obtained after the eigendecomposition of the linear Koop- man operator A can be interpreted as capturing low-dimensional structures underlying the evolu- tion of hidden representations. We interpret these structures to reflect linguistic and semantic reg- ularities, such as syntactic patterns, grammatical dependencies, and features relevant to the predic- tive behavior of the model. Apart from the basic DMD algorithm we also use the higher order dynamic mode decomposition (HODMD) (Le Clainche and Vega, 2017). While DMD focuses on finding a linear operator satis- fying the Markovian assumption (h t+1 = Ah t ), HODMD implements time embedding (see Ap- pendix C for more details). In this way, a Hankel matrix is constructed from the original snapshot data to model the system such that each hidden state is treated as a linear combination of thed previous time-steps, wheredis the delay parame- ter. To implement HODMD, we stackdsuccessive snapshots into a higher-dimensional representation, which allows the model to capture long-term dy- namics and temporal dependencies that standard DMD might miss. The expanded snapshot matrix H looks as follows H N−d 1 = h ℓ ∗ 1 h ℓ ∗ 2 ... h ℓ ∗ N−d h ℓ ∗ 2 h ℓ ∗ 3 ... h ℓ ∗ N−d+1 . . . . . . . . . . . . h ℓ ∗ d h ℓ ∗ d+1 ... h ℓ ∗ N (4) Our pipeline then proceeds by applying standard DMD to this augmented matrix. First, we per- form an SVD onHto reduce dimensionality while preserving dominant features. Next, we compute the Koopman eigenvalues and eigenvectors from the reduced-rank operator to identify the system’s modes. Mode ranking: Similar to PCA, where compo- nents are ranked by the amount of variance ex- plained, DMD/HODMD modes can be ranked ac- cording to different measures of dynamical signif- icance. We employ two ranking criteria: (i) the magnitude of the initial modal amplitudes (Row- ley et al., 2009), and (i) the average magnitude of the modal amplitudes over the temporal trajectory (Tissot et al., 2014) to get the most relevant modes (top-k) for our task. In all our experiments, we considerk = 5modes, both when experimenting with DMD (and other similar baselines like PCA). Token ranking: To find how each token con- tributes to those modes, we consider the state change (∆h ℓ ∗ t ) which the tokenw t adds after it has been processed, as a proxy for the hidden state representation of that token, and then calculate its scalar projection magnitude on each of the modes. We then sum the projection magnitude values for each mode for all the topkmodes to obtain a pro- jection score for that token. ∆h ℓ ∗ t = h ℓ ∗ t − h ℓ ∗ t−1 (5) α t,i = ⟨∆h ℓ ∗ t ,φ i ⟩ ∥φ i ∥ , i = 1,...,k(6) s t = k X i=1 α t,i (7) where,φ i is a selected mode,⟨·⟩represents the dot product of two vectors,krepresents the number of top modes considered, ands t is the projected score obtained for the particular token indext. Finally, we rank the tokens according to their decreasing order of projection scores to obtain the ranked input attributions for each token. 5 Experimental setup Models: In this work, we fine-tune three differ- ent LLMs,Llama-3.2-3B-inst(Grattafiori et al., 2024),Qwen3-4B-inst(Yang et al., 2025), and Mistral-7B-v0.3-inst(Jiang et al., 2023). We perform supervised instruction fine-tuning of the model in full precision. The prompt templates used for the fine-tuning are provided in the Appendix. Datasets: For this work, we use three text classifi- cation datasets (see Appendix D for more details) as follows – (i) Sentiment: sentiment classifica- tion (Zhang et al., 2015), (i) FakeEdit: fake news detection (Nakamura et al., 2020), and (i) Hat- eXplain: hateful content classification (Mathew et al., 2021). We use the training split of each of these datasets to perform the supervised fine-tuning of the three chosen models. Finally, we use 1000 different sentences from the test set of each dataset, equally divided in different input size buckets, to evaluate the efficacy of our method on small as well as large input sentences. Baselines: We compare our method with IG (Sun- dararajan et al., 2017), SHAP (Lundberg and Lee, 2017) (more specifically, the GRADIENTSHAP implementation) and PCA. Note that PCA is com- puted directly on the transpose of debiased hidden state matrix, ˆ X ⊤ , such that each row is treated as an observation (token hidden states) and each column represents the hidden dimension, to obtain a set of principal components that maximizes the variance in the collection of representation vectors. The token attribution in PCA then follows the same procedure as DMD, mentioned in Eq. 5 to 7. Metrics: We report three metrics: (i) average matched count, (i) Rank-Biased Overlap (RBO) (Webber et al., 2010) and (i) recall@k. The av- erage matched count gives the expected number of tokens matched from the GT, it does not take into account the number of tokens GT has for a particular sentence. RBO compares the rank of the retrieved token with the token’s rank in the GT and penalizes tokens that are assigned a higher rank than in the GT. RBO is defined as follows, RBO EXT (S,T,p,k) = (1−p) k X d=1 A d ·p d−1 + X k k ·p k (8) whereSandTare two retrieved lists,pis the persis- tence factor,A d is the agreement between two lists at depthd,kis the top-kcut-off value, andX k is the overlap count betweenSandTat depthk. For our evaluations, we choosep = 0.95. Recall@k measures the percentage of GT tokens retrieved. It is sensitive to both the number of retrieved tokens and the number of tokens in the GT. Ground truth: For each dataset, ground truth to- kens responsible for a prediction were obtained us- ing OpenAI GPT-4.1 (Achiam et al., 2023), where the model was prompted to rank individual tokens from input sentences based on their importance to the output label. A maximum of 20 tokens were ranked for each input, apart from stop words, and the model was instructed to stop the ranking early if there were fewer than 20 influential tokens. Table 2 lists an example sentence from each dataset’s class label, and the corresponding ground truth (GT) re- turned by GPT-4.1. The GTs are ranked in order of the most influential to the least influential in de- termining the class label of the input sentence. In order to test the goodness of the labelling, two ex- perienced annotators manually rank 100 input sen- tences from each dataset which is then compared Dataset classesInput sentenceGround truth (GT) Negative sentimenthated it it was so small not as the picture showed could not even get it to turn on i threw it in the trash it wasnt worth the hassle of returning it 1. hated, 2. trash, 3. hassle, 4. not, 5. small, 6. threw, 7. worth, 8. returning Positive sentimentgreat gift but expensive i got this gift from my husband it is a little expensive but the best gps so far 1. best, 2. great, 3. gift, 4. husband, 5. gps HateXplain (hateful)user it a refugees fault for sucking up all the oxygen thus my issues get ignored or minimized citizens before regugees 1. refugees, 2. fault, 3. sucking, 4. oxygen, 5. citizens, 6. regugees FakeEdit (fake)womens magazine staff out of work as kardashians reveal they have nothing left to reveal 1. kardashians, 2. reveal, 3. nothing, 4. left, 5. womens, 6. magazine, 7. staff, 8. out, 9. work Table 2: Examples of input sentences and their corresponding token-level ground truth (GT) rankings generated by GPT-4.1. with the GPT-4.1 annotated rankings. We first ob- serve that the two experienced annotators agree very well in terms of the RBO scores for all the three datasets. The scores are as follows – negative sentiment: 0.64, positive sentiment: 0.72, hateful text: 0.62, and fake text: 0.61. Next, we find that the human annotations and GPT-4.1 annotations also align well. In particular, RBO values obtained for the three datasets are negative sentiment: 0.60, positive sentiment: 0.70, hateful text: 0.54, and fake text: 0.60. The slightly lower agreement for HateXplain corresponds to the known subjectivity of hate speech annotation in the literature. 6 Results We evaluate our framework using 1,000 samples per dataset, partitioned into distinct buckets based on input length to assess performance across vary- ing context sizes. For the Sentiment dataset, sam- ples are categorized into three ranges: 15–40, 40–70, and 70–100 tokens. For the HateXplain and FakeEdit datasets, we utilize two buckets: 15–40 and 40–70 tokens. As noted earlier, we evaluate four combinations as follows – (i) standard DMD with amplitude-based ranking, (i) standard DMD with time-averaged amplitude ranking, (i) HODMD with amplitude- based ranking using an adaptive delay parameter, and (iv) HODMD with time-averaged amplitude ranking using an adaptive delay parameter. In each setting (model + dataset), we report the metrics ob- tained for the best performing combination above alongside the baselines (see Appendix D for full results). 6.1 Dataset based results Sentiment classification: For sentiment classifi- cation, we retrieve the top 20 tokens for both sen- timents and then compare them with the GT. We find that HODMD (see Table 3), paired with mode ranking through the averaged amplitude, consis- tently outperforms all other configurations across the three evaluated models. This setup generally exceeds traditional baselines, including PCA, IG, and SHAP in most metrics. Since sentiment is typ- ically expressed through multi-word phrases rather than isolated tokens, the use of delay factors in the HODMD setup provides the necessary context to capture these dynamics more effectively than the standard DMD. HateXplain:For this dataset, the standard DMD using amplitude-based ranking proves to be the most effective configuration for all mod- els.As each data point in HateXplain con- tains very few sentences with more than 10 GT tokens, we calculate the metrics for the top 10 retrieved tokens.Note that here we com- pute the attributions on for the class of interest (i.e., hateful class). ForLlama-3.2-3B-instand Qwen3-4B-inst, the DMD-based attribution gen- erally outperforms the baseline methods (Table 4). ForMistral-7B-v0.3-inst, our method is the second best. Since HateXplain relies on the spe- cific token-level annotations for hateful content, we believe that the state changes triggered by these individual tokens are well-captured by the stan- dard DMD. This suggests that the evolution of hidden representations in this context follows a near-Markovian process that does not require the extended memory of HODMD. FakeEdit: The results for FakeEdit align closely with those of HateXplain (Table 4). Because the dataset consists of Reddit posts where spe- cific keywords—such as the names of political fig- ures—often dictate the “fake” status of a post, the importance is concentrated on individual tokens. Here, again, the attributions are computed for the fake class. Consequently, the standard DMD with amplitude ranking achieves the best performance, outperforming all baseline methods in recall@10. The importance ranking of the tokens may not align best with the GT, but our method retrieves the most Method Negative sentimentPositive sentiment MC@20↑RBO@20↑Recall@20↑MC@20↑RBO@20↑Recall@20↑ Llama-3.2-3B-inst IG4.420.240.604.690.260.66 SHAP3.850.200.474.260.210.50 PCA5.240.240.665.980.260.73 DMDINTEL HODMD−avgamp 5.350.250.686.120.270.76 Qwen3-4B-inst IG4.600.250.575.240.280.63 SHAP3.940.200.484.260.210.50 PCA5.460.250.696.070.280.74 DMDINTEL HODMD−avgamp 5.430.260.696.170.290.76 Mistral-7B-v0.3-inst IG5.030.270.625.140.260.61 SHAP4.020.200.484.380.220.51 PCA5.230.240.655.720.250.70 DMDINTEL HODMD−avgamp 5.400.250.686.030.270.75 Table 3: Experimental results for the Sentiment dataset across three different model families. MC: Matched count, HODMD- avgamp: HODMD with averaged amplitude ranking. Best results are in bold and the second best are underlined. Method Hateful reviews (HateXplain)Fake reviews (FakeEdit) MC@10↑RBO@10↑Recall@10↑MC@10↑RBO@10↑Recall@10↑ Llama-3.2-3B-inst IG2.570.240.604.390.380.60 SHAP2.300.220.503.600.320.48 PCA2.630.230.605.120.430.70 DMDINTEL DMD−amp 2.770.250.645.460.420.76 Qwen3-4B-inst IG2.660.260.604.350.380.60 SHAP2.310.220.503.850.340.52 PCA2.630.240.604.900.410.68 DMDINTEL DMD−amp 2.850.250.645.370.420.75 Mistral-7B-v0.3-inst IG2.730.260.624.430.390.61 SHAP2.600.250.594.000.350.54 PCA2.950.300.675.210.440.72 DMDINTEL DMD−amp 2.820.260.655.270.410.74 Table 4: Experimental results for the HateXplain and FakeEdit datasets across three different model families. The attributions are computed only for the class of interest (i.e., hateful for HateXplain and fake for FakeEdit). MC: Matched count, DMD-amp: DMD with amplitude ranking. Best results are in bold and the second best are underlined. amount of tokens from GT as measured by recall. This confirms that for tasks where localized token information is critical, standard linear approxima- tions are highly effective. 6.2 Qualitative results Some of the representative qualitative results are noted in Table 5. DMDINTEL consistently demon- strates the strongest alignment with GT attributions across both models and sentence types, ranking the most semantically relevant tokens near the top of its lists. For the negative sentence, DMDINTEL places “trash” and “unable” within its top four for Llama-3.2-3B-inst, while for the hate speech it recovers all five GT tokens within its top nine for both models. Notably, DMDINTEL also exhibits strong cross-model consistency – its top five to- kens for hate speech are nearly identical between Llama-3.2-3B-instandQwen3-4B-inst, differ- ing only in minor reordering. SHAP performs competitively, recovering several high-priority GT tokens in compact lists: for the hate speech sen- tence it ranks “fa**ot” and “retarded” in its top two for both models, and for the negative sentence it places “trash” first forLlama-3.2-3B-inst. How- ever, SHAP tends to surface contextually plausible but GT-absent tokens (e.g. “high”, “conference”, “scheduled”) at the expense of key sentiment mark- ers such as “unable” and “shrieking”. PCA shows moderate but uneven alignment, capturing several GT tokens yet struggling to prioritize the most discriminative ones (e.g., ranking “ni**er”, the highest GT token, last in the hate speech sentence forLlama-3.2-3B-inst). IG shows the weakest ModelMethodTop-ranked Tokens (Attribution) “dont buy this phone this phone makes a constant high static shrieking noise i bought this phone to use on a regularly scheduled long conference call and was totally unable to use it its going directly in the trash" [Negative] Llama-3.2-3B-inst GT1. trash, 2. unable, 3. shrieking, 4. static, 5. noise, 6. constant, 7. dont, 8. buy, 9. totally, 10. directly DMDINTEL1. going, 2. trash, 3. dont, 4. unable, 5. conference, 6. shrieking, 7. makes, 8. bought, 9. regularly, 10. high, 11. scheduled, 12. use, 13. long, 14. directly, 15. phone, 16. constant, 17. call, 18. static, 19. totally PCA1. going, 2. unable, 3. conference, 4. shrieking, 5. makes, 6. high, 7. bought, 8. trash, 9. phone, 10. regularly, 11. constant, 12. static, 13. long, 14. use, 15. noise, 16. directly IG1. scheduled, 2. dont, 3. conference, 4. use, 5. totally, 6. trash, 7. directly, 8. going, 9. use, 10. static, 11. call, 12. unable, 13. noise, 14. buy, 15. regularly, 16. long SHAP1. trash, 2. directly, 3. noise, 4. high, 5. scheduled, 6. use, 7. constant, 8. going, 9. static, 10. dont, 11. phone, 12. totally Qwen3-4B-inst GT1. trash, 2. unable, 3. shrieking, 4. static, 5. noise, 6. constant, 7. dont, 8. buy, 9. totally, 10. directly DMDINTEL1. going, 2. directly, 3. trash, 4. dont, 5. regularly, 6. makes, 7. buy, 8. static, 9. unable, 10. bought, 11. scheduled, 12. use, 13. totally, 14. long, 15. phone, 16. shrieking, 17. conference, 18. constant, 19. noise PCA1. dont, 2. buy, 3. going, 4. bought, 5. makes, 6. regularly, 7. shrieking, 8. trash, 9. constant, 10. scheduled, 11. static, 12. noise, 13. directly, 14. use, 15. conference, 16. totally, 17. call, 18. long, 19. unable IG1. use, 2. dont, 3. unable, 4. regularly, 5. totally, 6. conference, 7. scheduled, 8. use, 9. noise, 10. directly, 11. going, 12. call, 13. constant, 14. phone, 15. bought, 16. high, 17. static, 18. phone, 19. trash SHAP1. use, 2. high, 3. conference, 4. totally, 5. dont, 6. regularly, 7. makes, 8. phone, 9. buy, 10. scheduled, 11. directly, 12. shrieking “you are retarded and do not know what reply mean or are a shitty troller suck a bag of ni**er d*cks fa**ot" [Hateful] Llama-3.2-3B-inst GT1. ni**er, 2. fa**ot, 3. retarded, 4. shitty, 5. d*cks DMDINTEL1. fa**ot, 2. d*cks, 3. suck, 4. ni**er, 5. bag, 6. troller, 7. shitty, 8. retarded, 9. mean PCA1. suck, 2. reply, 3. know, 4. retarded, 5. troller, 6. bag, 7. d*cks, 8. shitty, 9. mean, 10. fa**ot, 11. ni**er IG1. shitty, 2. fa**ot, 3. troller, 4. mean, 5. know, 6. bag, 7. ni**er SHAP1. fa**ot, 2. retarded, 3. know, 4. ni**ger, 5. bag, 6. shitty Qwen3-4B-inst GT1. ni**er, 2. fa**ot, 3. retarded, 4. shitty, 5. d*cks DMDINTEL1. fa**ot, 2. d*cks, 3. bag, 4. suck, 5. ni**er, 6. troller, 7. reply, 8. mean, 9. shitty, 10. know, 11. retarded PCA1. retarded, 2. d*cks, 3. fa**ot, 4. bag, 5. suck, 6. know, 7. reply, 8. mean, 9. troller, 10. shitty, 11. ni**er IG1. bag, 2. ni**er, 3. retarded, 4. fa**ot, 5. suck, 6. troller, 7. know, 8. d*cks, 9. shitty SHAP1. fa**ot, 2. retarded, 3. reply, 4. ni**er, 5. d*cks Table 5: Attribution for Llama-3.2-3B-inst and Qwen3-4B-inst models on negative sentiment and hate speech sentences. performance overall, with non-salient tokens fre- quently appearing at the top of its rankings and instability manifesting as duplicate token entries (e.g., “use” appears twice in itsQwen3-4B-inst negative sentence list). Beyond ranking quality, the methods also differ in coverage and consistency. DMDINTEL and PCA produce longer attribution lists, surfacing a broader set of contributing tokens, while SHAP and IG yield shorter and more concentrated lists. This compactness is a strength for SHAP, which tends to maintain reasonable precision within its smaller set, but a liability for IG, which risks omitting relevant tokens such as “noise” and “con- stant” in the negative sentence. Across models, PCA and IG exhibit greater sensitivity to the un- derlying model’s representations — for instance, PCA’s top-ranked tokens for the negative sentence differ substantially betweenLlama-3.2-3B-inst andQwen3-4B-inst— whereas DMDINTEL and SHAP remain comparatively stable. Taken to- gether, these observations suggest that DMDIN- TEL offers the best balance of GT recall, coverage, and cross-model robustness, with SHAP being a reasonable but less comprehensive alternative, and PCA and IG largely lagging behind. 6.3 Sensitivity to layer selection Here we investigate how sensitive DMDINTEL is to the layer selected (see Algorithm 1) for obtaining our results. We report the recall values for each of the datasets and models in Table 7. We note that DMDINTEL is quite stable, and the results using ℓ ∗ − 1orℓ ∗ + 1are very similar to those usingℓ ∗ . In fact, as long as the cosine similarity of the bias and the selected layer is close to that between the bias and ℓ ∗ , the results remain largely unchanged. Method Sentiment AnalysisHateXplain Negative SentimentPositive SentimentHateful Label Acc Drop (%)↑Conf Drop↑Acc Drop (%)↑Conf Drop↑Acc Drop (%)↑Conf Drop↑ Llama-3.2-3B-inst IG12.180.1515.170.2050.660.34 PCA13.680.1718.670.2456.880.37 DMDINTEL12.570.1819.260.2545.500.32 Qwen3-4B-inst IG15.770.1919.070.2457.010.37 PCA12.680.1926.150.2847.750.32 DMDINTEL10.780.1825.350.2844.310.31 Mistral-7B-v0.3-inst IG26.750.278.480.0834.790.22 PCA19.760.2020.260.1951.190.33 DMDINTEL 17.760.1921.660.2144.530.29 Table 6: Comparison of accuracy drop (%) and confidence score drop undertop-ktoken masking across the models on Sentiment and HateXplain datasets. Higher drops (↑) indicate more faithful attributions. ModelDatasetℓ ∗ − 1ℓ ∗ ℓ ∗ + 1∆ max Llama-3.2-3B-inst Negative0.690.680.680.01 Positive0.760.750.750.01 Hateful0.640.640.630.01 Fake0.760.760.760.00 Qwen3-4B-inst Negative0.680.690.680.01 Positive0.740.760.750.02 Hateful0.620.640.620.02 Fake0.750.750.750.00 Mistral-7B-v0.3-inst Negative0.670.670.670.00 Positive0.740.740.730.01 Hateful0.650.650.640.01 Fake0.730.740.740.01 Table 7: Layer-wise sensitivity analysis.∆ max represents the maximum DMD performance difference observed from layer ℓ ∗ to either ℓ ∗ − 1 or ℓ ∗ + 1. 6.4 Fidelity based analysis To evaluate the faithfulness of our input attribu- tion framework, we conduct a token-masking ex- periment following established perturbation bench- marks. Specifically, we measure the drop in model accuracy and output confidence when masking the top-kmost influential tokens as identified by each attribution method, with whitespace charac- ters. As reported in Table 6, our approach consis- tently ranks first or second in both accuracy and confidence drops across all evaluated model fam- ilies and datasets, confirming its robust capability to locate the most critical input features. Notably, while gradient-based baselines like IG achieve high accuracy drops on HateXplain by isolating explicit target terms, they often focus strictly on a narrow subset of toxic keywords. This behavior is corrobo- rated by higher RBO scores alongside lower recall values in Table 4, indicating that IG overlooks im- plicit, contextually essential tokens. In contrast, our method captures a broader, more cohesive set of influential features while maintaining competitive perturbation performance. 7 Conclusion In this paper, we proposed a framework for inter- preting the predictions of fine-tuned LLMs by an- alyzing the evolution of their hidden states during sequence processing. By treating the model’s inter- nal mechanics as a dynamical system, we demon- strate that DMD provides a more effective lens for input attribution than traditional methods. Our find- ings suggest that this dynamical perspective offers a better and more robust way to understand how LLMs represent and process complex information compared to state-of-the-art methods. Acknowledgment The authors gratefully acknowledge the financial and institutional support provided by the Joint PhD Programme between the Indian Institute of Tech- nology Kharagpur (IIT Kharagpur) and The Uni- versity of Manchester. 8 Limitations In this section, we discuss potential limitations of our approach in this work for interpreting LLMs. First, we only fine-tune and evaluate LLMs for clas- sification tasks, where there are very few tokens (1-2) generated as a response to the input query. While this makes our approach efficient for iden- tifying key tokens from the input when only one to two tokens are generated as output, it is still a challenge to find out the influential tokens from the input if more tokens are generated as these tokens also get fed to the model in an auto-regressive man- ner to generate the next token. For example, in ma- chine translation from English to French, one can find the tokens that dominate the low dimensional structures for the first few French tokens generated, but as these French tokens are also fed to the model to generate the complete translation, formulating the DMD matrix in this case becomes somewhat difficult. So accurately mapping the English tokens responsible for French translated tokens becomes a challenging task and is potential future work. Second, in this work we only explore supervised in- struction fine-tuned LLMs for simple downstream tasks, but the general purpose instruction models released by the companies remains unexplored in this study. These instruction tuned models are base models fine-tuned on huge datasets for instruction following tasks, and identifying key patterns in their input processing is also a potential future work. References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 techni- cal report. arXiv preprint arXiv:2303.08774. Nora Belrose, Igor Ostrovsky, Lev McKinney, Zach Fur- man, Logan Smith, Danny Halawi, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predic- tions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112. Nicola Demo, Marco Tezzele, and Gianluigi Rozza. 2018. PyDMD: Python dynamic mode decomposi- tion. Journal of Open Source Software, 3(22):530. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 conference of the North American chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, volume 1 (long and short papers), pages 4171– 4186. N Benjamin Erichson, Steven L Brunton, and J Nathan Kutz. 2019. Compressed dynamic mode decomposi- tion for background modeling. Journal of Real-Time Image Processing, 16(5):1479–1492. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5484–5495. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Maziar S Hemati, Clarence W Rowley, Eric A Deem, and Louis N Cattafesta. 2017. De-biasing the dy- namic mode decomposition for applied koopman spectral analysis of noisy datasets. Theoretical and Computational Fluid Dynamics, 31(4):349–368. Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2023. Linearity of relation decoding in transformer language models. arXiv preprint arXiv:2308.09124. Albert Q Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Guillaume Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv:2310.06825. Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and 1 others. 2020. Captum: A unified and generic model interpretability library for pytorch. arXiv preprint arXiv:2009.07896. J Nathan Kutz, N Benjamin Erichson, Travis Askham, Seth Pendergrass, and Steven L Brunton. 2017. Dy- namic mode decomposition for background model- ing. In Proceedings of the 16th IEEE International Conference on Computer Vision (ICCV), Venice, Italy, pages 22–29. Soledad Le Clainche and José M. Vega. 2017. Higher order dynamic mode decomposition. SIAM Journal on Applied Dynamical Systems, 16(2):882–925. Scott M Lundberg and Su-In Lee. 2017. A unified ap- proach to interpreting model predictions. Advances in Neural Information Processing Systems, 30. Shuiyang Mao, PC Ching, and Tan Lee. 2020. Eigen- emo: Spectral utterance representation using dy- namic mode decomposition for speech emotion clas- sification. arXiv preprint arXiv:2008.06665. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukher- jee. 2021. Hatexplain: A benchmark dataset for ex- plainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 35, pages 14867–14875. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associ- ations in gpt. Advances in Neural Information Pro- cessing Systems, 35:17359–17372. Kai Nakamura, Sharon Levy, and William Yang Wang. 2020. Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. In Pro- ceedings of the twelfth language resources and eval- uation conference, pages 6149–6157. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, and 1 oth- ers. 2022. In-context learning and induction heads. arXiv preprint arXiv:2209.11895. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, and 1 others. 2018. Improving language understanding by generative pre-training. Clarence W. Rowley, Igor Mezic, Shervin Bagheri, Phillpp Schlatter, and Dan S. Henningson. 2009. Spectral analysis of nonlinear flows. Journal of Fluid Mechanics, 641:115–127. S Sachin Kumar, M Anand Kumar, KP Soman, and Prabaharan Poornachandran. 2019. Dynamic mode- based feature with random mapping for sentiment analysis. In Intelligent Systems, Technologies and Applications: Proceedings of ISTA 2018, pages 1–15. Springer. Peter J Schmid. 2010. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics, 656:5–28. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Inter- national Conference on Machine Learning, pages 3319–3328. PMLR. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. Gilles Tissot, Laurent Cordier, Nicolas Benard, and Bernd R. Noack. 2014. Model reduction using Dy- namic Mode Decomposition. Comptes Rendus. Mé- canique, 342(6-7):410–416. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Process- ing Systems, 30. MT Vyshnav, Sachin Kumar, and KP Soman. 2020. Of- fensive language detection: A comparative analysis. arXiv preprint arXiv:2001.03131, 10. William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Trans. Inf. Syst., 28(4). An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025.Qwen3 technical report.arXiv preprint arXiv:2505.09388. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in Neural Information Processing Systems, 28. A Use of AI assistance We employed proprietary LLMs solely for obtain- ing the ground truth labels for the datasets, edito- rial purposes, including refining grammar, spelling, word choice, and overall clarity of the manuscript. B Dynamic mode decomposition Dynamic mode decomposition was introduced in (Schmid, 2010), to extract dynamic patterns from a flow field in fluid mechanics. DMD works on observational data, without any underlying assump- tions of the dynamical system. Consider a set of observations obtained from a dynamical system, represented by the matrix X N 1 , X N 1 = [x 1 ,x 2 ,...,x N ] where each column vectorx i represents ani th ob- servation of size m. Next, the assumption is that, there exists a linear mapping A which connects the observation x i to x i+1 as follows x i+1 = Ax i In the matrix notation we can write the above equa- tion as, X N 2 ≈ AX N−1 1 (9) where,X N−1 1 ∈ m×n−1 , is the matrix containing the firstN − 1snapshots, andX N 2 is the matrix shifted by one time-step. Generally, the spatial dimension of the systems is much larger than the temporal dimension,m ≫ nand hence Eq 9 can be solved using the singular-value decomposition (SVD): X = U ΣV ∗ (10) where * represents complex conjugate transpose, U ∈C n×r , Σ∈C r×r , and V ∈C r×n . Hence A = X N 2 V Σ −1 U ∗ (11) as computing this can be computationally expen- sive, we project A onto the lower dimensional sub- space defined by the eigenvectors of the snapshot matrix, ̃ A = U ∗ AUand then again define the DMD problem as, ̃ x t+1 = ̃ A ̃ x t (12) After computing ̃ A , the eigenvector-decomposition is done and then the lower dimensional eigenvec- tors are projected back to the original spatial di- mension, to approximate the eigenvectors of A. ̃ AW = WΛ(13) Φ = X N 2 V Σ −1 W(14) The eigenvectors obtained from Eq 13, are known as DMD modes, and they represent the dominant patterns which construct the flow of the dynamical system. Each eigenvectorφ i , is associated with a complex eigenvalue which represents the strength and the frequency of oscillations of that eigenvec- tor. C Higher order dynamic mode decomposition HODMD exploits a delay-embedding approach which helps it capture temporal dependencies across multiple snapshots. Consider a set of obser- vationsX N 1 = [x 1 ,x 2 ,...,x N ]. The core assump- tion of HODMD is that there exists a higher-order linear relationship such that the observationx i+d is a linear combination of the previous d snapshots: x i+d ≈ A 0 x i +A 1 x i+1 +·+A d−1 x i+d−1 (15) wheredis the design parameter representing the number of delays. This is solved by constructing a Hankel snapshot matrixH N−d+1 1 by stacking dsuccessive snapshots into a higher-dimensional representation as follows. H N−d+1 1 = x 1 x 2 ... x N−d+1 x 2 x 3 ... x N−d+2 . . . . . . . . . . . . x d x d+1 ... x N (16) In this augmented space, the system can be treated as a first-order dynamical system: H N−d+1 2 ≈AH N−d 1 (17) whereH N−d+1 2 is the Hankel matrix shifted by one time-step. To efficiently compute the operatorA, the SVD is performed on the Hankel matrix similar to DMD. H N−d 1 =U ΣV ∗ (18) The high-dimensional operator is then projected onto the lower-dimensional subspace defined by the SVD modes of the Hankel matrix, yielding the reduced-order operator ̃ A: ̃ A =U ∗ H N−d+1 2 VΣ −1 (19) The reduced operator is then eigendecomposed as: ̃ AW = WΛ(20) The HODMD modes are obtained by projecting these eigenvectors back. Since the eigenvectorsW exist in the augmentedm× ddimensional space, the spatial DMD modesΦare recovered by ex- tracting the firstmcomponents (the first block) of the projected eigenvectors: Φ =U 1:m,: W(21) Each modeφ i inΦrepresents a spatio-temporal pattern, while the corresponding eigenvalue inΛ describes the temporal evolution (frequency and growth/decay rate) of that specific pattern. D Experimental setting and detailed results We fine-tune all the models on a single NVIDIA H100 GPU, and we utilize NVIDIA L40 GPUs for model inference and execution of the input attribu- tion pipelines. D.1 Sentiment analysis We use Amazon polarity dataset which has reviews extracted from Amazon and their corresponding labels as positive and negative (Zhang et al., 2015). All the three LLMs used in this study, were fully fine-tuned on 150,000 train samples in full preci- sion. The test accuracy achieved on 10,000 test samples was around 97% for all the three models with very minor variations. The instruction prompt used for fine-tuning the LLMs and also used to get the bias vector without passing the input sentence for sentiment analysis is as follows. We use the chat template format provided by Hug- gingFace to define the above chat template, and then HuggingFace handles the tokenization of this template internally. To obtain the vector corre- sponding to the entire representation of the instruc- tion template, we find the review index _start and then use the index before that to extract the bias vector from the hidden states. Table 8 shows the comparison study of our DMD attribution pipeline of all possible ranking variants against the base- lines. From the metrics, it is evident that the modes whose amplitudes persist for longer, capture the sentiment better than those whose initial amplitude has a very high value. D.2 HateXplain From HateXplain (Mathew et al., 2021), we only consider attribution towards hateful sen- tences, although the model was trained for bi- nary classification with hateful and normal la- bels. After fine-tuning on 10,000 samples for 2 epochs,Llama-3.2-3B-instachieved an accu- racy of 82%,Qwen3-4B-instachieved an accuracy of 83.2% andMistral-7B-v0.3-instachieved an accuracy of 84% on the test set consisting of 2,000 samples. The instruction prompt used for fine-tuning the LLMs is as follows: The same prompt was also used for the generation of bias vector from the layer considered in the ex- periments. Table 8 shows full comparison study between different DMD algorithms and ranking techniques. As the dataset contains the hateful trig- gers as single words or a few collection of words, it can be well approximated by vanilla DMD, and the modes whose amplitude starts high prove to be useful for finding the correct attribution tokens. D.3 FakeEdit We use FakeEdit dataset, which contains posts taken from Reddit and categorized into two labels; fake and not-fake.These reviews are mostly post titles with short to medium input length sentences.We fine-tune the LLMs on a train set of 15,000 samples for 2 epochs and achieve an accuracy of 84% for Llama-3.2-3B-inst, 85.6% forQwen3-4B-inst and 86% forMistral-7B-v0.3-inston a test set of 2,000 samples. The prompt template used in fine-tuning and generating the bias vector is as fol- lows. The observations for FakeEdit are mostly similar to HateXplain, as FakeEdit also contains individ- ual trigger words which can determine the “fak- eness” of a sentence. Consequently, DMD with amplitude ranking performs the best among all vari- ants and baselines. D.4IG and SHAP We use Captum library (Kokhlikyan et al., 2020) to implement the pipelines for IG and SHAP. In both implementations, we calculate the attributions on the embedding layer for each token, which directly measures the impact of the input tokens on the generated output. E Analysis of eigenvalues and the modes The temporal evolution of the dynamical system is governed by the discrete-time eigenvaluesλ i ob- tained from the eigendecomposition of the linear operatorA. These eigenvalues provide critical in- formation regarding the stability, growth, and oscil- latory nature of their corresponding DMD modes φ i . The position of the eigenvalues relative to the unit circle in the complex plane determines the asymptotic behavior of the modes categorized as follows. • Steady modes (|λ i | = 1): Eigenvalues falling exactly on the unit circle correspond to pure oscillations with constant amplitude. These represent the steady-state dynamics of the sys- tem. •Stable modes (|λ i | < 1): Eigenvalues lo- cated within the unit circle represent phys- ically damped or transient dynamics. The closer the eigenvalue is to the origin, the more Table 8: Detailed Attribution results across models and datasets. Best results are noted in bold. Llama-3.2-3B-inst - Sentiment Method NegativePositive RBO RecRBO Rec IG0.230.580.240.66 SHAP0.150.330.170.38 PCA0.240.660.260.73 DMDINTEL DMD−amp 0.240.670.260.74 DMDINTEL DMD−avgamp 0.240.670.270.75 DMDINTEL HODMD−amp 0.240.680.270.75 DMDINTEL HODMD−avgamp 0.250.680.270.76 Qwen3-4B-inst - Sentiment Method NegativePositive RBO RecRBO Rec IG0.250.570.280.63 SHAP0.200.480.210.50 PCA0.250.690.280.74 DMDINTEL DMD−amp 0.240.680.270.74 DMDINTEL DMD−avgamp 0.240.680.280.76 DMDINTEL HODMD−amp 0.240.680.270.75 DMDINTEL HODMD−avgamp 0.260.690.290.76 Mistral-7B-v0.3-inst - Sentiment Method NegativePositive RBO RecRBO Rec IG0.270.670.260.61 SHAP0.200.480.220.51 PCA0.240.650.250.70 DMDINTEL DMD−amp 0.230.650.250.72 DMDINTEL DMD−avgamp 0.240.670.260.73 DMDINTEL HODMD−amp 0.240.660.260.74 DMDINTEL HODMD−avgamp 0.250.680.270.75 Llama-3.2-3B-inst - HateXplain and FakeEdit Method HatefulFake RBO RecRBO Rec IG0.240.600.380.60 SHAP0.220.500.320.48 PCA0.230.600.430.70 DMDINTEL DMD−amp 0.250.640.420.76 DMDINTEL DMD−avgamp 0.240.630.420.75 DMDINTEL HODMD−amp 0.250.640.410.74 DMDINTEL HODMD−avgamp 0.240.630.410.74 Qwen3-4B-inst - HateXplain and FakeEdit Method HatefulFake RBO RecRBO Rec IG0.260.600.380.60 SHAP0.220.500.340.52 PCA0.240.600.410.68 DMDINTEL DMD−amp 0.250.640.420.75 DMDINTEL DMD−avgamp 0.240.620.410.74 DMDINTEL HODMD−amp 0.250.630.410.74 DMDINTEL HODMD−avgamp 0.240.620.410.73 Mistral-7B-v0.3-inst - HateXplain and FakeEdit Method HatefulFake RBO RecRBO Rec IG0.260.620.390.61 SHAP0.250.590.350.54 PCA0.300.670.440.72 DMDINTEL DMD−amp 0.260.650.410.74 DMDINTEL DMD−avgamp 0.260.640.410.74 DMDINTEL HODMD−amp 0.250.630.400.73 DMDINTEL HODMD−avgamp 0.250.630.400.73 rapid the decay of the mode as the sequence progresses. •Unstable modes (|λ i | > 1): Eigenvalues outside the unit circle indicate exponential growth, often associated with diverging hid- den state representations or numerical insta- bilities in the local flow. Based on this theory, we conduct an analysis of the eigenvalues and the dynamics of the hidden states of the representative sentences from the test set of all datasets for each model. We find that for all the sentences, the hidden states produced by the model induce decaying and stable modes. While some modes persist forever without any oscillations (mode 0 in the plots), there exist some modes that decay to zero amplitude before the sentence even finishes, and some modes still oscillate without ever converging to zero amplitude. Figure 2 provides a comprehensive comparison of eigenvalue distributions and mode dynamics across various model and dataset configurations. Each sub- figure consists of two complementary plots: the left plot displays the unit circle (with real and imagi- nary components on the axes), while the right plot illustrates the corresponding temporal dynamics. In the dynamics plot, thex-axis represents the token index (time-step) and they-axis tracks the real part of the mode’s amplitude. These dynamics plots ef- fectively show how the influence of specific modes fluctuates throughout the processing of a sentence. To ensure a fair comparison, we use the same sam- ple input sentences for every model-dataset pair. The results reveal that even when processing iden- tical inputs, the selected modes and their behaviors vary significantly across models. This divergence is likely due to differences in hidden state dimensions, which force each model to encode and process in- formation in its own distinct way. This observation is further supported by the results in Table 8 of this Appendix. Although PCA performance fluctuates significantly between different models even on the same dataset, our optimized DMD configuration remains remarkably stable. This consistency sug- gests that DMD is less sensitive to architectural variations, providing more reliable attribution met- rics across diverse model scales and hidden state dimensions. F Error analysis We also conduct an error analysis of how good is the approximationAfound by DMD by recon- structing the hidden states for sample sentences and then measuring the Frobenius norm between the original hidden states and the reconstructed states. The reconstruction of the snapshot matrix ˆ X orig is achieved by utilizing the extracted DMD modes Φ, the diagonal matrix of eigenvaluesΛ, and the vector of initial amplitudesα. The approximated state at time t is given by: ˆ h t = r X i=1 φ i λ t−1 i α i = ΦΛ t−1 α(22) For the entire sequence ofNsnapshots, the recon- structed matrix ˆ X recons is represented as: ˆ X recons = Φdiag(α) 1 λ 1 ... λ N−1 1 1 λ 2 ... λ N−1 2 . . . . . . . . . . . . 1 λ r ... λ N−1 r (23) To evaluate the fidelity of this approximation, we calculate the relative reconstruction error using the Frobenius norm||·|| F . The errorεis defined as the ratio of the norm of the residual to the norm of the original snapshot matrix: ε = || ˆ X orig − ˆ X recons || F || ˆ X orig || F (24) Table 9 represents the error values for the mod- els and the datasets. We find that the reconstruction error rate is very low in the order of10 −2 , suggest- ing that the linear approximation of the LLM using DMD is quite accurate. G Computational efficiency and resource utilization To assess the computational efficiency of our pro- posed framework, we benchmark its average exe- cution time per input and peak GPU memory foot- print against established baselines. All timing and memory profiling experiments were conducted on NVIDIA L40 GPUs using single-precision floating- point format (float32) for model parameters and internal vector representations. As detailed in Ta- ble 10, our approach achieves superior efficiency, requiring the lowest processing time and mini- mal GPU memory overhead. This performance gain primarily stems from our architectural design: while standard gradient- and perturbation-based techniques like IG and SHAP require dozens to hundreds of forward and backward passes per in- put sequence, our method extracts attributions non- (a) Llama-3.2-3B-inst: Negative sentiment(b) Qwen3-4B-inst: Negative sentiment (c) Mistral-7B-v0.3-inst: Negative sentiment(d) Llama-3.2-3B-inst: Positive sentiment (e) Qwen3-4B-inst: Positive sentiment(f) Mistral-7B-v0.3-inst: Positive sentiment (g) Llama-3.2-3B-inst: HateXplain(h) Qwen3-4B-inst: HateXplain (i) Mistral-7B-v0.3-inst: HateXplain(j) Llama-3.2-3B-inst: FakeEdit (k) Qwen3-4B-inst: FakeEdit(l) Mistral-7B-v0.3-inst: FakeEdit Figure 2: Eigenvalue distributions and mode dynamics across various model and dataset configurations. Label Reconstruction error (ε) Llama-3.2-3B-inst Qwen3-4B-inst Mistral-7B-v0.3-inst Negative Sentiment0.0700.0740.059 Positive Sentiment0.0690.0740.060 Hateful Label0.0960.0730.059 Fake Label0.1320.0990.121 Table 9: Reconstruction error ε. ModelMethod Sentiment AnalysisHateXplainFakeddit GPU (GB)↓Time (s)↓GPU (GB)↓Time (s)↓GPU (GB)↓Time (s)↓ Llama-3.2-3B-inst IG15.23± 1.209.07 / 8.2414.70± 0.506.1714.18± 0.125.86 SHAP38.80± 3.002.62 / 2.3035.302.2127.22± 4.201.66 PCA12.901.10 / 1.1412.990.8712.960.73 DMDINTEL (Ours)13.011.03 / 1.1012.900.3312.970.36 Qwen3-4B-inst IG18.20± 0.9010.98 / 10.0218.15± 0.305.9617.785.74 SHAP45.30± 4.203.37 / 3.5639.56± 3.202.3440.15± 2.11.97 PCA16.181.61 / 1.6516.071.0016.220.79 DMDINTEL (Ours)16.31± 0.100.87 / 0.9616.130.4016.220.37 Mistral-7B-v0.3-inst IG28.26± 1.3016.30 / 16.8230.09± 0.2011.4029.86± 0.2011.30 SHAP– PCA28.261.73 / 2.2128.221.0428.220.94 DMDINTEL (Ours)28.281.03 / 1.6528.220.5128.220.47 Table 10: Detailed efficiency evaluation comparing peak GPU memory usage (GB) and per-sample attribution computation time (seconds) across model families and benchmark datasets. For Sentiment dataset, execution times are reported for negative/positive context passes. intrusively via a single forward pass. Consequently, this yields a substantial reduction in inference la- tency and resource consumption. H Ablation studies H.1 Results without denoising To assess the importance of removing the noise inflicted by the empty chat prompt template, we studied the metrics without the removal. We find that the metrics drop significantly for sentiment analysis, there is a slight drop for HateXplain but the drop in metrics is not so significant for Fa- keEdit. Detailed results for the three models and the datasets for DMDINTEL without the noise re- moval are presented in the Tables 11 and 12. This reinforces that the noise removal step not only re- moves the noise, but also ensures that the hidden state snapshots contain the information which the specific token adds to the residual stream. H.2 Threshold τ = 0.5 Raising the similarity threshold to0.5restricts layer selection strictly to representations that exhibit a higher alignment with the model’s instruction-following dynamics. Intuitively, enforc- ing a stricter cutoff filters out lower-level contex- tual signals, which is expected to yield a corre- sponding decline in attribution performance met- rics. Our empirical observations largely align with this expectation, albeit with domain-specific nu- ances. For the FakeEdit dataset, we observe a mi- nor reduction across evaluation metrics. On HateX- plain, performance remains largely stable, accom- panied by a subtle increase in RBO scores. In con- trast, for the Sentiment dataset across both positive and negative sentiment classes, raising the thresh- old induces a pronounced drop in metric values forQwen3-4B-instandMistral-7B-v0.3-inst, whereasLlama-3.2-3B-instmaintains relatively resilient performance. Complete results across all configurations are reported in Tables 13 and 16. H.3 Varying the k value in top-k tokens for calculation of metrics We check the consistency of the results when we select top-10ranked tokens with the GT for the Sen- timent dataset, and top-20 ranked tokens with GT for FakeEdit and HateXplain datasets. We find that the results remain consistent in the Sentiment dataset, while in that of FakeEdit and HateXplain the metrics start to converge for top-20ranked to- kens as there are very few input sentences from these datasets which have GT tokens more than 10. Model Negative SentimentPositive Sentiment MC@20↑RBO@20↑Recall@20↑MC@20↑RBO@20↑Recall@20↑ Llama-3.2-3B-inst5.190.230.665.920.260.73 Qwen3-4B-inst5.280.230.675.990.260.73 Mistral-7B-v0.3-inst 5.130.230.645.810.250.71 Table 11: Evaluation results of our proposed method (DMDINTEL HODMD−avgamp ) on Sentiment dataset across Negative and Positive sentiment classes for three model families without denoising the hidden states. Model Hateful Reviews (HateXplain)Fake Reviews (FakeEdit) MC@10↑RBO@10↑Recall@10↑MC@10↑RBO@10↑Recall@10↑ Llama-3.2-3B-inst2.760.240.635.450.420.75 Qwen3-4B-inst2.800.240.635.370.410.75 Mistral-7B-v0.3-inst2.800.260.645.270.410.74 Table 12: Evaluation results of our proposed method (DMDINTEL DMD−amp ) on the HateXplain and FakeEdit datasets across three model families without denoising the hidden states. Detailed results are presented in Tables 14 and 15. H.4 Selecting top-7 modes We examine the effect of mode subspace dimen- sionality by projecting token representations onto the top-7modes compared to top-5in our original setup. As reported in Table 17, selecting additional modes leads to a noticeable drop in performance metrics for DMDINTEL. This performance de- crease likely stems from noisy higher-order modes that introduce non-label-intensive tokens into the top attributions. In contrast, PCA shows slight, though statistically marginal, performance gains when the number of modes is increased. MethodMCRBORec@k Llama-3.2-3B-inst – Hateful (HateXplain) DMDINTEL DMD−amp 2.790.250.64 PCA2.710.250.62 Llama-3.2-3B-inst – Fake (FakeEdit) DMDINTEL DMD−amp 5.360.410.74 PCA5.110.410.70 Qwen3-4B-inst – Hateful (HateXplain) DMDINTEL DMD−amp 2.850.260.64 PCA2.800.260.63 Qwen3-4B-inst – Fake (FakeEdit) DMDINTEL DMD−amp 5.330.410.74 PCA4.920.410.68 Mistral-7B-v0.3-inst – Hateful (HateXplain) DMDINTEL DMD−amp 2.810.260.64 PCA2.880.290.66 Mistral-7B-v0.3-inst – Fake (FakeEdit) DMDINTEL DMD−amp 5.170.400.72 PCA4.870.410.68 Table 16: Comparative evaluation results for HateX- plain and FakeEdit datasets, with threshold τ = 0.5. I Label-intensive token alignment vs. decision faithfulness To systematically evaluate the semantic relevance of identified tokens, we employ GPT-4.1 to anno- tate input sequences with label-oriented ground- truth rationale tokens. For any LLM fine-tuned on downstream classification, these label-intensive to- kens intuitively represent the core semantic anchors driving the model’s target class selection. Our em- pirical findings demonstrate that DMD success- fully isolates the low-dimensional latent subspaces onto which these label-intensive tokens yield high projection values, revealing their prominent role within the model’s implicit representation dynam- ics. Crucially, we distinguish between semantic coverage and exclusive causal dependency. Iden- tifying these dominant tokens does not imply that the model relies solely on them to synthesize its output, nor does it suggest that non-labeled con- textual tokens are irrelevant to the decision tra- jectory. As evidenced by our fidelity-by-masking experiments (Section 6.4)—particularly on the Ha- teXplain dataset where gradient-based baselines like IG exhibit sharper accuracy and confidence drops—DMD prioritizes broad, contextually rich label-intensive patterns over narrow, isolated toxic keywords. Consequently, while IG aggressively de- grades task performance by removing single high- leverage tokens, our approach captures a more com- prehensive, contextually aligned feature set that reflects the broader latent mechanisms governing sequence processing. Method Top20 NegativePositive Matched countRBORecall@kMatched countRBORecall@k Llama-3.2-3B-inst DMDINTEL HODMD−avgamp 5.290.240.676.020.260.74 PCA 5.310.250.675.910.260.72 Qwen3-4B-inst DMDINTEL HODMD−avgamp 5.180.230.655.830.250.72 PCA5.090.240.645.600.230.69 Mistral-7B-v0.3-inst DMDINTEL HODMD−avgamp 5.160.230.645.780.250.71 PCA 5.000.230.625.460.230.68 Table 13: Comparative evaluation results for Sentiment Analysis across negative and positive classes with threshold τ = 0.5. Method Negative sentimentPositive sentiment MC@10↑RBO@10↑Recall@10↑MC@10↑RBO@10↑Recall@10↑ Llama-3.2-3B-inst IG2.930.260.393.090.280.45 SHAP2.520.210.332.700.230.38 PCA3.190.260.423.510.280.45 DMDINTEL HODMD−avgamp 3.280.260.433.730.290.48 Qwen3-4B-inst IG3.170.290.413.550.310.45 SHAP2.730.220.353.000.240.37 PCA3.350.270.443.700.300.48 DMDINTEL HODMD−avgamp 3.430.280.453.800.310.49 Mistral-7B-v0.3-inst IG3.410.310.433.450.290.43 SHAP2.770.230.353.010.240.37 PCA3.130.250.403.260.260.41 DMDINTEL HODMD−avgamp 3.400.270.443.650.290.47 Table 14: Results for top-10 ranked tokens in case of Sentiment dataset. Method Hateful reviews (HateXplain)Fake reviews (FakeEdit) MC@20↑RBO@20↑Recall@20↑MC@20↑RBO@20↑Recall@20↑ Llama-3.2-3B-inst IG3.340.200.735.310.310.70 SHAP2.690.170.583.730.240.50 PCA4.090.200.896.940.360.92 DMDINTEL DMD−amp 4.170.220.917.003.430.93 Qwen3-4B-inst IG3.640.220.784.880.300.65 SHAP2.750.170.594.060.250.54 PCA4.110.210.896.880.350.92 DMDINTEL DMD−amp 4.200.210.906.940.340.93 Mistral-7B-v0.3-inst IG3.340.210.725.120.310.69 SHAP3.120.200.674.230.260.57 PCA4.190.250.906.890.370.92 DMDINTEL DMD−amp 4.150.230.906.930.340.93 Table 15: Results for HateXplain and FakeEdit for top-20 ranked tokens. Method Negative sentimentPositive sentiment MC@20↑RBO@20↑Recall@20↑MC@20↑RBO@20↑Recall@20↑ Llama-3.2-3B-inst PCA5.280.240.665.990.270.73 DMDINTEL HODMD−avgamp 5.210.230.666.040.260.74 Qwen3-4B-inst PCA5.460.250.686.080.280.75 DMDINTEL HODMD−avgamp 5.340.240.676.100.270.75 Mistral-7B-v0.3-inst PCA5.270.240.665.760.250.71 DMDINTEL HODMD−avgamp 5.180.230.655.880.250.72 Table 17: Selecting top-7 modes for calculating the attribution for sentiment analysis dataset.