Paper deep dive

Gradient Atoms: Unsupervised Discovery, Attribution and Steering of Model Behaviors via Sparse Decomposition of Training Gradients

J Rosser

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 28

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%

Last extracted: 3/22/2026, 5:12:52 AM

Summary

Gradient Atoms is an unsupervised method for discovering, attributing, and steering model behaviors by decomposing training gradients into sparse components via dictionary learning in a preconditioned eigenspace. It identifies shared update directions (atoms) that correspond to interpretable task-type behaviors, which can then be used as steering vectors to control model outputs without requiring behavioral labels or per-query scoring.

Entities (5)

Dictionary Learning · technique · 100%EKFAC · algorithm · 100%Gemma-3 4B IT · model · 100%Gradient Atoms · method · 100%Training Data Attribution · field · 100%

Relation Signals (4)

Gradient Atoms → discovers → Model Behaviors

confidence 100% · Gradient Atoms, an unsupervised method that discovers candidate model behaviors from training gradients alone

Gemma-3 4B IT → finetunedvia → LoRA

confidence 100% · We use Gemma-3 4B IT fine-tuned via LoRA

Gradient Atoms → functionsas → Steering Vectors

confidence 100% · These atoms double as effective steering vectors: applying them as weight-space perturbations produces large, controllable shifts in model behavior

Gradient Atoms → uses → EKFAC

confidence 100% · We extract per-document gradients, project them into a preconditioned eigenspace (via EKFAC)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Training data attribution (TDA) methods ask which training documents are responsible for a model behavior. However, models often learn broad concepts shared across many examples. Moreover, existing TDA methods are supervised -- they require a predefined query behavior, then score every training document against it -- making them both expensive and unable to surface behaviors the user did not think to ask about. We present Gradient Atoms, an unsupervised method that decomposes per-document training gradients into sparse components ("atoms") via dictionary learning in a preconditioned eigenspace. Each atom captures a shared update direction induced by a cluster of functionally similar documents, directly recovering the collective structure that per-document methods do not address. Among 500 discovered atoms, the highest-coherence ones recover interpretable task-type behaviors -- refusal, arithmetic, yes/no classification, trivia QA -- without any behavioral labels. These atoms double as effective steering vectors: applying them as weight-space perturbations produces large, controllable shifts in model behavior (e.g., bulleted-list generation 33% to 94%; systematic refusal 50% to 0%). The method requires no query--document scoring stage, and scales independently of the number of query behaviors of interest. Code is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

27,642 characters extracted from source content.

Expand or collapse full text

Gradient Atoms GRADIENT ATOMS: UNSUPERVISED DISCOVERY,ATTRIBUTIONAND STEERING OF MODEL BEHAVIORS VIA SPARSE DECOM- POSITION OF TRAINING GRADIENTS J Rosser FLAIR, University of Oxford jrosser@robots.ox.ac.uk ABSTRACT Training data attribution (TDA) methods ask which training documents are respon- sible for a model behavior. However, models often learn broad concepts shared across many examples. Moreover, existing TDA methods are supervised—they require a predefined query behavior, then score every training document against it—making them both expensive and unable to surface behaviors the user did not think to ask about. We present Gradient Atoms, an unsupervised method that decomposes per-document training gradients into sparse components (“atoms”) via dictionary learning in a preconditioned eigenspace. Each atom captures a shared update direction induced by a cluster of functionally similar documents, directly recovering the collective structure that per-document methods do not address. Among 500 discovered atoms, the highest-coherence ones recover in- terpretable task-type behaviors—refusal, arithmetic, yes/no classification, trivia QA—without any behavioral labels. These atoms double as effective steering vectors: applying them as weight-space perturbations produces large, controllable shifts in model behavior (e.g., bulleted-list generation 33%→94%; systematic refusal 50%→0%). The method requires no query–document scoring stage, and scales independently of the number of query behaviors of interest. Code is available at https://github.com/jrosseruk/gradient_atoms. 1INTRODUCTION Training data attribution (TDA) attributes a model’s behavior to the documents it was trained on (Koh & Liang, 2017; Grosse et al., 2023; Bae et al., 2024). Per-document attribution has proven valuable for identifying influential examples, debugging predictions, and curating datasets. However, per-document scoring captures only part of what training teaches. A model that learns to perform arithmetic during fine-tuning does so not because of any single arithmetic example, but because many arithmetic examples induce a shared update direction across weights. More broadly, Ruis et al. (2024) show that fine-tuning primarily instils procedural capabilities—task-level strategies such as classification, editing, or code generation—that emerge from the collective gradient signal of functionally similar documents. Standard TDA also faces a practical bottleneck: it is supervised, requiring the user to specify a query behavior before scoring documents against it. This demands (1) knowing what behaviors to look for in advance, and (2) anO(N )scoring pass per query. For influence-function methods like EK- FAC (Grosse et al., 2023; Bae et al., 2024), attributingQbehaviors costsO(Q×N )query–document comparisons. For a comprehensive audit of learned behaviors, this is prohibitively expensive. We address both gaps with a different question. Instead of asking “which document caused this behavior?”, we ask: what are the shared update directions that clusters of documents jointly induce? 1 arXiv:2603.14665v2 [cs.AI] 17 Mar 2026 Gradient Atoms 500 atoms from sparse dictionary learning on 5,000 training gradients · Dot size coherence · Top 500 labeled 5 atoms with coherence > 0.5, 43 with coherence > 0.1 Gradient Atoms: Unsupervised Discovery of Model Behaviors Trivia Q&A Grammar editing Yes/No classification Arithmetic Multi-label classification Sentence transformation Sentence restructuring Multi-step math Code + translations Name-an-example Sentiment classification Entity fact answers Short phrase answers Refusal (missing input) Science/math facts Code generation Grammar correction Concise direct answers Generic/inspirational Word-level tasks Creative short writing Explanatory answers Long-form generation Comparison/analysis Step-by-step instructions List generation Email/letter drafting Persuasive writing Structured output Analogy/metaphor Informational answers Summarisation Dialogue responses Math word problems Numeric computation Casual grammar fix SQL queries Systematic refusal Token extraction Python functions Bulleted lists Numbered lists SQL + regex Code exec + classify Code (multi-lang) DB/web code Vocabulary tasks Summarise/paraphrase Numeric recall General knowledge 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Coherence Figure 1: Gradient atoms discovered via sparse dictionary learning over 5,000 training-document gradients. Each point is one atom; high-coherence atoms correspond to tightly defined task types. This is the idea behind Gradient Atoms. We extract per-document gradients, project them into a preconditioned eigenspace (via EKFAC), and apply sparse dictionary learning to decompose them into atoms. Each atom is a direction in weight space shared by a cluster of functionally similar documents. The method is: •Unsupervised: no behavioral labels, measurement functions, or contrastive pairs are needed. •Efficient: one decomposition surfaces candidate behaviors simultaneously, with no per- query scoring stage. • Actionable: each atom can be unprojected into a full-parameter steering vector. We validate this in two stages. First, we show that sparse dictionary learning on EKFAC- preconditioned gradients discovers 500 atoms, of which the highest-coherence atoms correspond to interpretable task types (Table 4). Second, we show that these atoms—discovered without any behavioral labels—function as effective steering vectors: applying them as weight perturbations produces large, controllable shifts in five tested behaviors (Figure 2). Our contributions are: 1.We show that per-document TDA provides an incomplete account of fine-tuning and propose a richer unit of analysis: shared update directions in gradient space. 2.We introduce Gradient Atoms, an unsupervised method that discovers candidate model behaviors from training gradients alone, without supervision or per-query scoring. 3.We demonstrate that discovered atoms function as effective steering vectors, producing large controllable shifts in model behavior without any behavioral labels. 2 Gradient Atoms 2RELATED WORK Several concurrent works share our core insight that gradient similarity reflects functional similarity among training examples. GradientSpace (Sridharan et al., 2025) clusters LoRA gradients to identify “latent skills” but uses them to train specialised expert routers, not for interpretability or steering. Mode-Conditioning (Wu et al., 2025) confirms that gradient clustering reliably recovers functional groupings (98.7% F1), but applies this to test-time compute allocation. ELROND (Skier ́ s et al., 2026) is philosophically closest: they decompose per-sample gradients into steerable directions via PCA and sparse autoencoders, but in diffusion models rather than LLMs. Our contribution combines: (1) operating on training gradients rather than inference activations, (2) being fully unsupervised with no per-query scoring, and (3) the discovered atoms directly functioning as steering vectors for LLMs. Standard TDA methods (Koh & Liang, 2017; Grosse et al., 2023; Bae et al., 2024) require a predefined query andO(Q× N )query–document comparisons; atoms complement this with unsupervised discovery. In the activation-space literature, SAEs (Nanda et al., 2023) and gradient-informed variants (Olmo et al., 2025; Shu et al., 2025) decompose activations at inference time; atoms decompose what the model learned during training, across all layers simultaneously. Extended discussion is in Section A. 3METHOD Consider a language model with parametersθfine-tuned on a dataset ofNinput–output pairs x 1 ,...,x N . Each documentx i induces a gradientg i = ∇ θ L CE (θ;x i ) ∈R d , wheredis the number of trainable parameters. The gradientg i is the direction the model’s weights would move to improve on documenti. Gradient Atoms decomposes these per-document gradients into sparse components; the highest-quality components isolate interpretable learned behaviors. The pipeline has five steps. 3.1PER-DOCUMENT GRADIENT EXTRACTION For each training documentx i , we compute the gradient of cross-entropy loss with respect to all trainable parameters, producing a gradient matrixG ∈R N×d . Documents that require similar computations tend to produce similar gradient vectors. 3.2EKFAC PROJECTION AND PRECONDITIONING The raw gradient space is anisotropic—some directions have high curvature (small weight changes cause large loss changes) and others have low curvature. Without correction, any decomposition is dominated by high-curvature directions, drowning out semantic structure. We use the EKFAC eigendecomposition (Grosse et al., 2023; Bae et al., 2024) of the approximate Fisher information matrix to correct for this. For each modulemwith eigenvectorsQ m and eigenvaluesλ m , we project each gradient into the top-k eigenvectors and precondition: ̃g (m) i = Q (k)⊤ m g (m) i ,ˆg (m) i = ̃g (m) i q λ (k) m + ε (1) After concatenating across allMmodules, the projected gradient isˆg i ∈R k total wherek total = k×M. This makes the space approximately isotropic: a unit step in any direction corresponds to a roughly equal change in loss, encouraging atoms to capture functionally distinct directions rather than curvature artifacts. 3.3SPARSE DICTIONARY LEARNING We normalize each projected gradient to unit norm (so atoms reflect direction, not magnitude) and apply sparse dictionary learning to decompose: ˆg i ≈ K X j=1 α ij d j (2) 3 Gradient Atoms whereD = [d 1 ,...,d K ]∈R K×k total are the atoms andα ij are sparse coefficients—most are zero, so each document is explained by a few atoms. The sparsity penalty encourages each atom to capture a single pattern rather than blending multiple unrelated behaviors. 3.4COHERENCE SCORING For each atomj, we identify its activating documents—those with non-zero coefficientα ij —and compute a coherence score over the top-n activating documentsS j : coherence(j) = 1 |S j |(|S j |− 1) X a̸=b∈S j cos(g a ,g b )(3) whereg a ,g b are the raw (unprojected, fulld-dimensional) gradients. High coherence suggests the atom has found a shared computational motif in the original weight space, rather than an artifact of the projection. 3.5UNPROJECTION TO STEERING VECTORS Any atom can be converted back to a full parameter-space vector by reversing the projection: v j = unproject(d j )∈R d (4) This vector can be applied as a weight-space perturbationθ new = θ± α· v j , analogous to curvature- aware model editing (Ikram et al., 2026). The key difference is thatv j was discovered unsupervised from the training data, rather than derived from a hand-crafted measurement function or contrastive pair. 4EXPERIMENTS 4.1SETUP Model. We use Gemma-3 4B IT fine-tuned via LoRA (rank 8) on theqprojandvprojmatrices across all 34 layers, yielding 2.2M trainable parameters across 136 LoRA modules. Dataset. The model is fine-tuned on 5,000 instruction–response pairs sampled from a general- purpose SFT mixture covering arithmetic, grammar correction, classification, code generation, QA, creative writing, and other tasks. EKFAC factors (eigendecomposition of the approximate Fisher) are computed on the full training set. Gradient Atoms pipeline. We extract per-document gradients for all 5,000 training examples, project via EKFAC into 6,800 dimensions (50 eigencomponents×136 modules, a328×reduction), and run MiniBatchDictionaryLearning (scikit-learn) withK = 500atoms and sparsity penalty α = 0.1. Coherence is computed over the top-20 activating documents per atom using the raw 2.2M-dimensional gradients. 4.2ATOM DISCOVERY From 500 atoms: 5 have coherence> 0.5, 43 have coherence> 0.1, and 457 have coherence< 0.1. Figure 1 shows the distribution. The top 5 atoms (coherence> 0.5) are: short factual QA (0.725), grammar editing (0.672), yes/no classification (0.647), simple arithmetic (0.643), and multi-category classification (0.614). The full top 50 are listed in Table 4. Atoms capture task types, not topics. The decomposition clusters training data by how the model responds (arithmetic, classification, editing, code) rather than what it responds about (science, history, culture). This is consistent with the procedural-knowledge hypothesis of Ruis et al. (2024) discussed in Section 1, and confirms that gradient structure is organised around shared computational strategies. High coherence = stereotyped tasks. The top-5 atoms are all formulaic task types whose activating documents have highly similar gradients, suggesting similar computational pathways. 4 Gradient Atoms Multiple granularities. Grammar correction appears three times (ranks 2, 17, 36) and code generation five times (ranks 16, 37, 40, 45, 46), at decreasing coherence. The dictionary finds sub-clusters that may reflect different sentence complexity levels or programming language families. Format atoms. Bulleted lists (#469) and numbered lists (#299) are separate atoms, suggesting the model uses distinct weight pathways for these formatting patterns. Refusal is discoverable. Two atoms (#52, #161) capture the model’s tendency to reply “Please provide the input” when task instructions lack content—a behavior learned from training data that appears to be separable from other behaviors in gradient space. Effect of sparsity penalty. Table 1 shows the effect ofα. Atα = 0.01, atoms are too dense (median ∼2500docs each), blending unrelated patterns. Atα = 0.1, atoms are selective (∼100docs each). At α = 1.0, the penalty overwhelms reconstruction and all coefficients are zero. Table 1: Effect of sparsity penalty α on atom quality. αDocs per atom (median)Atoms coh > 0.5Atoms coh > 0.1 0.01∼25003∼20 0.1∼100543 1.0000 4.3BEHAVIORAL STEERING A key test of whether gradient atoms capture genuine computational structure is whether they can steer model behavior when applied as weight-space perturbations. We select five atoms spanning a range of coherence scores and behavioral types, unproject each to a full LoRA parameter-space vector, and apply perturbationsθ new = θ±α·v j withα∈0.5, 1.0, 2.0, 5.0, 10.0in both directions. Since dictionary learning assigns atom signs arbitrarily, we test both. For each atom, we design 100 evaluation questions (∼60that naturally invite the target behavior,∼40neutral controls) and measure behavior with a regex detector: • Yes/No (#415): first line starts with Yes/No/True/False • Code (#64): response contains a fenced code block (‘) • Refusal (#161): matches clarification-seeking patterns • Bullets (#469): ≥ 2 lines starting with -, * , or• • Numbered (#299): ≥ 2 lines starting with +[.)] Each steered adapter is served via vLLM alongside the clean baseline, with all 11 variants (5 alphas × 2 signs + baseline) loaded simultaneously as LoRA modules. 0.512510 0 20 40 60 80 100 % Detected base=39% #415 Yes/No Classification (coh=0.647) 0.512510 0 20 40 60 80 100 base=42% #64 Code Generation (coh=0.201) 0.512510 0 20 40 60 80 100 base=50% #161 Systematic Refusal (coh=0.111) 0.512510 0 20 40 60 80 100 base=33% #469 Bulleted List Generation (coh=0.103) 0.512510 0 20 40 60 80 100 base=58% #299 Numbered List Generation (coh=0.103) Behavioral Atom Steering: Alpha Sweep (Both Directions) Toward (v)Away ( +v) Figure 2: Behavioral steering via unsupervised gradient atoms. Red bars show the “toward” direction (θ− αv), blue bars show “away” (θ + αv), and the dashed line marks the clean baseline. Four of five atoms produce large, monotonic steering effects in at least one direction. Results are shown in Figure 2 and Table 2. All five atoms steer behavior in at least one direction, and four produce large effects (>14p): 5 Gradient Atoms Table 2: Behavioral atom steering results. For each atom, we report the baseline detection rate and the best increase/decrease achieved across all alpha values and both directions.∆is the change in percentage points from baseline. AtomBehaviorCoh.BaseBest↑∆↑Best↓∆↓ #415Yes/No Classification0.64739%51%+12p0% −39p #64Code Generation0.20142%58%+16p28% −14p #161Systematic Refusal0.11150%55%+5p0% −50p #469Bulleted Lists0.10333%94%+61p0% −33p #299Numbered Lists0.10358%59%+1p8% −50p •Bulleted lists (#469) is the strongest: steering increases bullet usage from 33% to 94% (+61p) and suppresses it to 0%. The effect is monotonic with alpha in both directions. • Refusal (#161) is completely suppressible: from 50% baseline to 0% atα = 5. The steered model responds “Okay.” to underspecified prompts instead of asking for clarification. The reverse direction modestly increases refusal (+5p) and makes the model more verbose, suggesting this atom captures a terse/verbose dimension. • Code generation (#64) increases from 42% to 58% (+16p) or decreases to 28% (−14p). The sign is flipped relative to the Newton step convention, confirming that atom signs from dictionary learning are arbitrary. •Yes/No (#415) shows moderate amplification (+12p) but strong suppression (−39p to 0%). At high alpha, the model loses coherence, establishing the upper bound of useful perturbation. •Numbered lists (#299) is asymmetric: easily suppressed (58%→8%) but not amplifiable (+1p), likely due to a ceiling effect. 5DISCUSSION Suppression appears easier than amplification. All five atoms suppress their target behavior to near zero, but only two (bullets, code) achieve substantial amplification. One interpretation is that suppressing a behavior requires disrupting a single computational pathway, while amplifying requires strengthening it against many competing alternatives. Coherence does not necessarily predict steerability. Atom #469 (coherence 0.103) produces the largest steering effect (+61p), while #415 (coherence 0.647) gives only+12p. Coherence measures gradient alignment among activating documents, but steerability also depends on how much the model’s default behavior already saturates the target pathway. Limitations. Only 43 of 500 atoms have coherence> 0.1, the majority are noise or capture overly broad mixtures. The instruction-following training data means atoms recover task types rather than fine-grained semantic preferences; more naturalistic data might yield different atoms. The 6,800-dim EKFAC projection discards information, and 5,000 documents may not cover rare behaviors. Our regex-based evaluation measures surface formatting rather than deeper behavioral changes. 6CONCLUSION We presented Gradient Atoms, an unsupervised method that discovers what a fine-tuning dataset teaches a model by decomposing training gradients into sparse components. The method addresses a gap in standard training data attribution: rather than scoring individual documents against a known behavior, it recovers the shared update directions that clusters of documents jointly induce—the collective structure through which procedural capabilities are acquired. The highest-coherence atoms recover interpretable task-type behaviors without supervision, and function as effective steering vectors, connecting unsupervised behavior discovery with controllable model editing. Future directions include composing multiple atoms for simultaneous multi-behavior steering, scaling the dictionary to 1,000+ atoms, cross-model comparison to identify shared vs. adapter-specific behaviors, and developing principled methods for alpha selection. 6 Gradient Atoms ACKNOWLEDGEMENTS J Rosser is supported by the EPSRC centre for Doctoral Training in Autonomous and Intelligent Machines and Systems EP/Y035070/1. Special thanks to the London Initiative for Safe AI and Arcadia Impact for providing workspace. REFERENCES Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Grosse. Training data attribution via approximate unrolled differentiation, 2024. URL https://arxiv.org/abs/2405.12186. RDWS Cook et al. Residuals and influence in regression. 1982. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023. Zarif Ikram, Arad Firouzkouhi, Stephen Tu, Mahdi Soltanolkotabi, and Paria Rashidinejad. Crispedit: Low-curvature projections for scalable non-destructive llm editing, 2026. URLhttps://arxiv. org/abs/2602.15823. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, p. 1885–1894. PMLR, 2017. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023. Jeffrey Olmo, Jared Wilson, Max Forsey, Bryce Hepner, Thomas Vin Howe, and David Wingate. Features that make a difference: Leveraging gradients for improved dictionary learning, 2025. URL https://arxiv.org/abs/2411.10397. Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rockt ̈ aschel, Edward Grefenstette, and Max Bartolo. Procedural knowl- edge in pretraining drives reasoning in large language models. arXiv preprint arXiv:2411.12580, 2024. Dong Shu, Xuansheng Wu, Haiyan Zhao, Mengnan Du, and Ninghao Liu. Beyond input activations: Identifying influential latents by gradient sparse autoencoders, 2025. URLhttps://arxiv. org/abs/2505.08080. Paweł Skier ́ s, Tomasz Trzci ́ nski, and Kamil Deja. Elrond: Exploring and decomposing intrinsic capabilities of diffusion models, 2026. URL https://arxiv.org/abs/2602.10216. Shrihari Sridharan, Deepak Ravikumar, Anand Raghunathan, and Kaushik Roy. Gradientspace: Unsupervised data clustering for improved instruction tuning, 2025. URLhttps://arxiv. org/abs/2512.06678. George Wang and Daniel Murfet. Patterning: The dual of interpretability, 2026. URLhttps: //arxiv.org/abs/2601.13548. Chen Henry Wu, Sachin Goyal, and Aditi Raghunathan. Mode-conditioning unlocks superior test-time scaling, 2025. URL https://arxiv.org/abs/2512.01127. 7 Gradient Atoms AEXTENDED RELATED WORK Training data attribution. Influence functions (Cook et al., 1982; Koh & Liang, 2017) estimate the effect of removing a training example on predictions. Grosse et al. (2023) scaled these to LLMs via EKFAC, and Bae et al. (2024) improved accuracy with approximate unrolled differentiation. All such methods are supervised—requiring a query behavior andO(N )scoring per query. Gradient Atoms complements these approaches by discovering candidate behaviors without a predefined query. Gradient-based clustering. GradientSpace (Sridharan et al., 2025) clusters LoRA gradients via online SVD to identify “latent skills,” sharing our core insight that gradient similarity reflects functional similarity. However, they use clusters to train specialised LoRA experts with a router, not for interpretability or steering. Their SVD + k-means yields routing labels; our sparse dictionary learning produces individually steerable atoms. Mode-Conditioning (Wu et al., 2025) independently confirms that gradient clustering recovers functional groupings (98.7% F1), applying this to test-time compute allocation. Gradient decomposition in diffusion models. ELROND (Skier ́ s et al., 2026) decomposes per- sample gradients via PCA and sparse autoencoders into steerable directions for visual attribute control—the diffusion-model analogue of our approach. Key differences: they decompose gradients within a single prompt’s realisations rather than across the full training set, and target visual attributes rather than LLM behaviors. Activation-space interpretability. SAEs decompose single-layer activations into monosemantic features (Nanda et al., 2023); gradient-informed variants (g-SAEs (Olmo et al., 2025), GradSAE (Shu et al., 2025)) use gradients to improve feature selection. Both still decompose activations—we decompose the gradients themselves, across all layers simultaneously. Wang & Murfet (2026) frame the theoretical dual of interpretability (behavior→training causes); Gradient Atoms provides a practical mechanism that additionally discovers behaviors without a query. Model editing and procedural knowledge. Prior steering methods require a known concept with measurement functions or contrastive pairs (Ikram et al., 2026); Gradient Atoms discovers steering directions unsupervised. Our finding that atoms capture task types rather than topics is consistent with Ruis et al. (2024), who show procedural knowledge drives what models extract from training data. BCOMPUTATIONAL DETAILS Table 3: Computational cost of the Gradient Atoms pipeline. StepResourcesTime Gradient extraction8× A100 40GB170s EKFAC projectionCPU, 16GB RAM ∼5 min Dictionary learning (α = 0.1)CPU, 32GB RAM ∼15 min Coherence computationCPU, 8GB RAM ∼5 min Total∼25 min Model: Gemma-3 4B IT, LoRA rank 8 (qproj+vproj), 2.2M parameters, 136 modules across 34 layers. EKFAC factors computed separately on the full training set. CFULL ATOM TABLE 8 Gradient Atoms Table 4: Top 50 gradient atoms ranked by coherence score. Each atom was characterised by manual inspection of its top-20 activating documents. RankAtomCoherenceActive DocsDescription 1#3480.725139Short factual Q&A—trivia with one-word/numeric answers 2#3280.672110Grammar and sentence editing 3#4150.647156Yes/No/True/False binary classification 4#4580.643124Simple arithmetic 5#4980.614176Multi-category classification and labeling 6#3580.49988Sentence transformation (voice, tense, translation) 7#20.463206Sentence restructuring (questions, passive/active) 8#4510.395182Multi-step arithmetic and unit conversions 9#4840.29849Mixed technical (code + translations + set ops) 10#3190.262180“Name an example of X”—single-entity retrieval 11#4300.258150Sentiment and text classification 12#4250.257215Single-entity factual answers 13#3630.238146Short phrase answers to open questions 14#520.23057“Please provide the input”—refusal on missing input 15#3640.205158Science and math fact answers 16#640.20125Code generation (Python, JS, C++, HTML) 17#3030.18949Grammar correction on short sentences 18#3940.188144Concise direct answers (mixed tasks) 19#4880.187168Short inspirational/generic responses 20#4770.185227Word-level tasks (synonyms, antonyms, rhymes) 21#3760.17697Creative short-form writing 22#1360.165161Multi-sentence explanatory answers 23#660.15445Long-form generation (essays, paragraphs) 24#4570.15283Comparison and analysis tasks 25#2560.15250Step-by-step instructions and how-to guides 26#2650.151118List generation (brainstorming, idea lists) 27#2240.14931Email and letter drafting 28#4460.14686Persuasive/argumentative writing 29#2940.14250Data extraction and structured output 30#4190.14278Analogy and metaphor reasoning 31#3590.137211Neutral informational answers 32#3060.136118Summarisation 33#1810.11987Dialogue and conversational responses 34#720.11837Math word problems 35#4450.11769Numeric computation (GCF, LCM, time) 36#4650.11670Grammar correction on casual sentences 37#2310.11521SQL queries and structured code 38#1610.11147Systematic refusal on unclear input 39#3250.106143Single-word/token extraction from input 40#610.10521Python utility function implementations 41#4690.103143Bulleted list generation 42#2990.10346Numbered list generation 43#670.1029SQL + regex + technical expressions 44#1800.10052Mixed code execution and classification 45#3810.09779Code generation (broad, multi-language) 46#4280.09681Database/web code (SQL, HTML, CSS, APIs) 47#480.09556Single-word vocabulary tasks (fill-in-blank, plurals) 48#4750.08883Summarisation and paraphrasing 49#2330.08746Numeric/factual recall with approximation 50#1720.084129General knowledge Q&A 9

Instant research discovery