Paper deep dive

nnterp: A Standardized Interface for Mechanistic Interpretability of Transformers

Clément Dumas

Year: 2025Venue: Mechanistic Interpretability Workshop at NeurIPS 2025Area: Mechanistic Interp.Type: ToolEmbeddings: 22

Models: Gemma, GPT-2, LLaMA, Mistral, Qwen

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/12/2026, 5:20:57 PM

Summary

nnterp is a lightweight library that provides a standardized interface for mechanistic interpretability of transformer models by wrapping NNsight. It addresses the fragmentation in HuggingFace model naming conventions and implementation details, enabling researchers to write consistent intervention code across 50+ model variants and 16 architecture families while preserving original model behavior and providing built-in validation.

Entities (5)

HuggingFace Transformers · software-library · 100%Mechanistic Interpretability · research-field · 100%NNSight · software-library · 100%nnterp · software-library · 100%TransformerLens · software-library · 100%

Relation Signals (3)

nnterp → standardizesaccessto → HuggingFace Transformers

confidence 100% · nnterp enables researchers to write intervention code once and deploy it across 50+ model variants

nnterp → wraps → NNSight

confidence 100% · nnterp, a lightweight wrapper around NNsight

nnterp → competeswith → TransformerLens

confidence 90% · Current approaches face a fundamental tradeoff: custom implementations like TransformerLens... To bridge this gap, we develop nnterp

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Mechanistic interpretability research requires reliable tools for analyzing transformer internals across diverse architectures. Current approaches face a fundamental tradeoff: custom implementations like TransformerLens ensure consistent interfaces but require coding a manual adaptation for each architecture, introducing numerical mismatch with the original models, while direct HuggingFace access through NNsight preserves exact behavior but lacks standardization across models. To bridge this gap, we develop nnterp, a lightweight wrapper around NNsight that provides a unified interface for transformer analysis while preserving original HuggingFace implementations. Through automatic module renaming and comprehensive validation testing, nnterp enables researchers to write intervention code once and deploy it across 50+ model variants spanning 16 architecture families. The library includes built-in implementations of common interpretability methods (logit lens, patchscope, activation steering) and provides direct access to attention probabilities for models that support it. By packaging validation tests with the library, researchers can verify compatibility with custom models locally. nnterp bridges the gap between correctness and usability in mechanistic interpretability tooling.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

22,068 characters extracted from source content.

Expand or collapse full text

nnterp: A Standardized Interface for Mechanistic Interpretability of Transformers Clément Dumas MATS, École Normale Supérieure Paris-Saclay, Université Paris-Saclay clement.dumas@ens-paris-saclay.fr Abstract Mechanistic interpretability research requires reliable tools for analyzing trans- former internals across diverse architectures. Current approaches face a funda- mental tradeoff: custom implementations like TransformerLens ensure consistent interfaces but require coding a manual adaptation for each architecture, introduc- ing numerical mismatch with the original models, while direct HuggingFace ac- cess through NNsight preserves exact behavior but lacks standardization across models. To bridge this gap, we develop nnterp, a lightweight wrapper around NNsight that provides a unified interface for transformer analysis while preserving original HuggingFace implementations. Through automatic module renaming and comprehensive validation testing, nnterp enables researchers to write interven- tion code once and deploy it across 50+ model variants spanning 16 architecture families. The library includes built-in implementations of common interpretabil- ity methods (logit lens, patchscope, activation steering) and provides direct access to attention probabilities for models that support it. By packaging validation tests with the library, researchers can verify compatibility with custom models locally. nnterp bridges the gap between correctness and usability in mechanistic inter- pretability tooling. 1 1 Introduction Mechanistic interpretability research aims to reverse-engineer the computational mechanisms within neural networks [Elhage et al., 2021, Olah et al., 2020]. For transformer language models, this re- quires tools that can reliably access and modify internal representations across diverse architectures. However, the field faces a fundamental engineering challenge: balancing implementation correct- ness with practical usability across the rapidly expanding landscape of transformer variants. Two dominant paradigms have emerged. TransformerLens [Nanda and Bloom, 2022] provides a clean, unified interface by reimplementing transformer architectures from scratch. This en- sures consistent module naming and reliable hooks but requires manual implementation for each new architecture, may introduce subtle differences from original models, and cannot leverage architecture-specific optimizations. Conversely, NNsight [Fiotto-Kaufman et al., 2024] operates directly on HuggingFace implementations, preserving exact model behavior and supporting any architecture HuggingFace provides. However, this approach inherits the fragmentation of Hug- gingFace’s naming conventions—accessing layers requires model.transformer.h for GPT-2 but model.model.layers for LLaMA—and leaves researchers vulnerable to breaking changes in transformer implementations. For example, since HugginFace transformers 4.54, Qwen and Llama 1 Repository available at https://github.com/Butanium/nnterp, documentation at https:// butanium.github.io/nnterp/. Install with pip install nnterp. 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: Mechanistic Inter- pretability. arXiv:2511.14465v2 [cs.LG] 14 Dec 2025 StandardizedTransformer Module Structure and Accessor Methods Modules embed_tokens layers[i] input self_attn Self-Attention input output mlp Feed-Forward input output output ln_final lm_head Accessor Methods model.logits→ get model.mlps_output[i]→ get/set model.mlps_input[i]→ get/set model.mlps[i]→ module model.attentions_output[i]→ get/set model.attentions_input[i]→ get/set model.attentions[i]→ module model.layers_output[i]→ get/set model.layers_input[i]→ get/set model.layers[i]→ module model.token_embeddings→ get/set Example Usage model =StandardizedTransformer("gpt2") # Access modules layer = model.layers[5] attn =model.attentions[5] norm = model.ln_final head =model.lm_head # Activations & interventions (in trace) withmodel.trace("Hello"): # Get activations (respect execution order) layer_in = model.layers_input[5] mlp_out = model.mlps_output[5] model .logits # Set activations (intervention) model.attentions_output[6] =... model.mlps_input[6] =... # Builtin interventions p = model.project_on_vocab(mlp_out) model .skip_layer(7) model .steer([8,9], vec, factor=0.5) All *_input and *_output accessors support both get and set for activation patching Module / Module accessorI/O accessor Figure 1: nnterp provides a standardized interface for transformer models. Left: It renames transformer modules to a consistent naming scheme (layers, self_attn, mlp, etc.). Middle: It provides accessor methods for internal activations, with get and set for all *_input and *_output. This handles whether the module returns a tuple or a single tensor. Right: Example usage for intervention and analysis. layers return activation tensors instead of tuples, which caused silent bugs in many interpretability experiments. This fragmentation creates significant friction. Researchers must either commit to a single tool’s limitations or maintain parallel codebases for different architectures. As mechanistic interpretability increasingly emphasizes cross-model validation and scalability to large models requiring optimiza- tions, neither approach alone suffices. We present nnterp, a lightweight library that bridges this gap by providing a standardized inter- face atop NNsight’s HuggingFace integration. Through systematic module renaming and valida- tion, nnterp enables researchers to write model.layers_output[5] consistently across GPT-2, LLaMA, Gemma, and other architectures while preserving exact HuggingFace behavior. Key con- tributions include: • A unified API for accessing transformer internals (layers, attention, MLP outputs) that works identically across 50+ model variants from 16 architecture families. • Automatic validation upon model loading that verifies module shapes and intervention cor- rectness. This caught the HuggingFace transformers 4.54 bug mentioned earlier day-1. • Built-in implementations of common interpretability methods (logit lens, patchscope, steer- ing) that work across all supported models. • Direct access to attention probabilities for compatible architectures through NNsight’s in- termediate variable tracking. • Packaged test suite enabling local verification of custom models. 2 Background and Related Work Mechanistic Interpretability Tooling. Early mechanistic interpretability relied on manual Py- Torch hooks to intercept activations [Elhage et al., 2021]. As the field matured, specialized libraries emerged to streamline common operations. Pyvene [Wu et al., 2024] provides a declarative frame- 2 work for causal interventions, while TransformerLens [Nanda and Bloom, 2022] offers a unified interface through custom implementations. NNsight [Fiotto-Kaufman et al., 2024] takes a different approach, wrapping existing models with a tracing system that enables interventions while preserv- ing original implementations. The Standardization Challenge. HuggingFace Transformers [Wolf et al., 2020] has become the de facto repository for transformer implementations, but its design prioritizes flexibility over con- sistency. Each architecture follows its own naming conventions and internal structure. This hetero- geneity reflects genuine architectural differences but complicates systematic analysis. Recent work on model diffing [Lindsey et al., 2024] and circuit discovery [Conmy et al., 2023] highlights the need for tools that work reliably across architectures. 3 nnterp Design and Implementation nnterp extends NNsight’s LanguageModel class with a StandardizedTransformer that au- tomatically renames modules to follow a consistent naming scheme and provides some I/O ac- cessors as shown in Figure 1. The module renaming system is implemented using the rename argument from the NNsight class, which allow to pass a dictionary of original names and new names. Different architectures require different renaming strategies. GPT-2’s transformer.h, attn, transformer.ln_f becomes layers, self_attn, ln_final, while LLaMA’s structure just moves model.layers to layers. nnterp maintains a configuration system that maps each architecture class to its renaming rules: m o d e l = S t a n d a r d i z e d T r a n s f o r m e r ( " g p t 2 " ) # I n t e r n a l l y a p p l i e s : r e n a m e = " t r a n s f o r m e r " : " m o d e l " , " h " : " l a y e r s " , " m o d e l . l a y e r s " : " l a y e r s " , " a t t n " : " s e l f _ a t t n " , " t r a n s f o r m e r . l n _ f " : " l n _ f i n a l " nnterp also provides model.layers/mlps/attentions_input/output[layer_idx] which allow to get and set the input and output of the specified layer. This should be preferred over using e.g. model.layers[layer_idx].output as this can be a tensor or a tuple depending on the architecture, while model.layers_output[layer_idx] always get/set the output tensor. This standardization leverages NNsight’s module renaming capability while maintaining full com- patibility with NNsight’s intervention API (the modules are still available under their original names). Researchers can use nnterp’s simplified interface for common operations or drop down to raw NNsight for advanced use cases. Attention probabilities access. To enable access to attention probabilities – which requires to use the slower eager attention implementation – enable_attention_probs=True needs to be passed to the StandardizedTransformer class. Once enabled, the attention probabilities can be accessed and set using the model.attention_probabilities[layer_idx] property. This relies on NNsight’s source feature, which allows to access intermediate variables in the forward pass. This means, that this is very implementation sensitive, and may break with new HuggingFace releases, e.g. if the name of the attention variable changes. The attention probabilities hookpoint is therefore not available on all models. See Appendix A for how to add support for new models. Validation guarantees. Upon initialization, nnterp runs automatic tests to verify: (1) module outputs have expected shapes, (2) attention probabilities sum to 1, (3) interventions affect outputs, and (4) layer skip operations preserve causality. These tests catch common issues like incorrect module identification or incompatible attention implementations. The validation suite ships with the package, allowing researchers to test custom models locally via python -m nnterp run_tests. 4 Extra features Built-in interventions. nnterp implements common interpretability methods that work across all supported models. Logit Lens [nostalgebraist, 2020] projects hidden states through the unembed- ding to see intermediate predictions. Patchscope [Ghandeharioun et al., 2024] replaces activations 3 from one context into another. Activation Steering adds steering vectors at specified layers. All methods use the same unified API across architectures. Prompt Management. nnterp provides a Prompt class that allows to track probabilities of specific target tokens in different categories. The easiest way to create a Prompt is to use the Prompt.from_strings method, which takes a prompt string and a dictionary of categories and target strings. The tracked tokens of each category is the set of all first tokens of the target strings with and without beginning of word. E.g for ["London", "Lyon"], the tracked tokens could be ["_London", "Lon", "_Lyo", "Ly"]. p r o m p t = P r o m p t . f r o m _ s t r i n g s ( " The c a p i t a lo fF r a n c ei s " , " t a r g e t " : " P a r i s " , " f a k e " : [ " London " , " Lyon " ] , t o k e n i z e r ) r e s u l t s = r u n _ p r o m p t s ( model , [ p r o m p t ] ) # R e t u r n s p r o b a b i l i t i e s f o r e a c h c a t e g o r y " t a r g e t " : t o r c h . T e n s o r ( [ 0 . 7 ] ) , " f a k e " : t o r c h . T e n s o r ( [ 0 . 1 ] ) 5 Empirical Validation nnterp supports 21 architecture families 2 . nnterp adds minimal overhead to NNsight’s already efficient implementation. NNsight’s performance analysis [Fiotto-Kaufman et al., 2024] shows it matches or exceeds TransformerLens speed while using less memory. Since nnterp is a thin wrap- per providing only interface standardization, it inherits these performance characteristics. 6 Discussion and Future Work Limitations. nnterp’s validation tests provide sanity checks rather than formal correctness guar- antees. While they catch common issues, subtle bugs in attention probability hooks or module identification may persist. The library also inherits NNsight’s limitations, including incompatibility with some attention implementations (e.g., Flash Attention for attention probabilities). Impact on Research Workflow. By separating interface standardization from implementation details, nnterp enables more reproducible mechanistic interpretability research. Researchers can share intervention code that works across models, facilitating replication and cross-architecture val- idation. The packaged test suite ensures that even custom models can be verified for compatibility. Future Directions. Future work includes automated architecture detection, support for non-causal and encoder-decoder architectures, access to attention KQV and MLP intermediate activations and access to MoE router’s logits. We are also exploring integration with NNsight itself and how to support remote execution via NDIF, which allow researcher to run their NNsight experiments on remote machines. 7 Conclusion nnterp demonstrates that the tradeoff between implementation correctness and interface usability in mechanistic interpretability tooling is not fundamental. By leveraging NNsight’s model wrapping capabilities and adding systematic validation, we provide researchers with both reliable access to exact HuggingFace implementations and a consistent interface across architectures. As mechanistic interpretability scales to larger models and broader architectural diversity, tools that balance correct- ness with usability become essential. nnterp represents a step toward more robust and reproducible interpretability research. 2 See Appendix B for the full list. 4 References Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer-circuits.pub/2021/framework/index.html. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits.Distill, 2020.doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in. Neel Nanda and Joseph Bloom. Transformerlens. https://github.com/TransformerLensOrg/ TransformerLens, 2022. Jaden Fiotto-Kaufman, Alexander R Loftus, Eric Todd, Jannik Brinkmann, Caden Juang, Koyena Pal, Can Rager, Aaron Mueller, Samuel Marks, Arnab Sen Sharma, Francesca Lucchetti, Michael Ripa, Adam Belfki, Nikhil Prakash, Sumeet Multani, Carla Brodley, Arjun Guha, Jonathan Bell, Byron Wallace, and David Bau. Nnsight and ndif: Democratizing access to foundation model internals. arXiv, 2024. URL https://arxiv.org/abs/2407.14561. Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christopher Manning, and Christopher Potts. pyvene: A library for understanding and improving PyTorch models via interventions. In Kai-Wei Chang, Annie Lee, and Nazneen Rajani, edi- tors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations), pages 158–165, Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.naacl-demo.16. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, 2020. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https: //aclanthology.org/2020.emnlp-demos.6/. Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread, 2024. URL https://transformer-circuits.pub/2024/crosscoders/index.html. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Ad- vances in Neural Information Processing Systems, volume 36, 2023. URL https://arxiv. org/abs/2304.14997. nostalgebraist. interpreting GPT: the logit lens. LessWrong, August 2020. URL https://w. lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models. In Forty-first International Conference on Machine Learning, 2024. URL https://arxiv.org/abs/2401. 06102. A Adding Support for Custom Models nnterp uses a standardized naming convention to provide a unified interface across transformer architectures. When a model doesn’t follow the expected naming patterns, researchers can use RenameConfig to map custom module names to the standardized interface. 5 A.1 Target Structure nnterp expects all models to follow this structure: S t a n d a r d i z e d T r a n s f o r m e r | − − e m b e d _ t o k e n s | − − l a y e r s [ i ] || − − s e l f _ a t t n |‘ − − mlp | − − l n _ f i n a l ‘ − − l m _ h e a d Models are automatically renamed to match this pattern using built-in mappings.The li- brary provides convenient accessors: layers_input[i], layers_output[i] for layer I/O, attentions[i], attentions_input[i], attentions_output[i] for attention, and mlps[i], mlps_input[i], mlps_output[i] for MLP components. A.2 Basic RenameConfig Usage When automatic renaming fails, create a custom RenameConfig specifying module names in the original model: f r o m n n t e r p i m p o r t S t a n d a r d i z e d T r a n s f o r m e r f r o m n n t e r p . r e n a m e _ u t i l s i m p o r t R e n a m e C o n f i g r e n a m e _ c o n f i g = R e n a m e C o n f i g ( m o d e l _ n a m e = " c u s t o m _ t r a n s f o r m e r " , l a y e r s _ n a m e = " c u s t o m _ l a y e r s " , a t t n _ n a m e = " c u s t o m _ a t t e n t i o n " , mlp_name = " c u s t o m _ f f n " , l n _ f i n a l _ n a m e = " c u s t o m _ n o r m " , l m _ h e a d _ n a m e = " c u s t o m _ h e a d " ) m o d e l = S t a n d a r d i z e d T r a n s f o r m e r ( " y o u r − model −name " , r e n a m e _ c o n f i g = r e n a m e _ c o n f i g ) For nested modules, use dot notation (e.g., layers_name="transformer.encoder_layers"). Multiple alternative names can be provided as lists (e.g., attn_name=["attention", "self_attention"]). A.3 Implementing Attention Probabilities Attention probabilities support requires implementing AttnProbFunction. Here’s the GPT-J im- plementation: f r o m n n t e r p . r e n a m e _ u t i l s i m p o r t A t t n P r o b F u n c t i o n c l a s s G P T J A t t n P r o b F u n c t i o n ( A t t n P r o b F u n c t i o n ) : d e f g e t _ a t t e n t i o n _ p r o b _ s o u r c e ( s e l f , a t t e n t i o n _ m o d u l e , r e t u r n _ m o d u l e _ s o u r c e = F a l s e ) : i f r e t u r n _ m o d u l e _ s o u r c e : r e t u r n a t t e n t i o n _ m o d u l e . s o u r c e . s e l f _ _ a t t n _ 0 . s o u r c e r e t u r n a t t e n t i o n _ m o d u l e . s o u r c e . s e l f _ _ a t t n _ 0 . s o u r c e \ . s e l f _ a t t n _ d r o p o u t _ 0 m o d e l = S t a n d a r d i z e d T r a n s f o r m e r ( " y u j i e p a n / g p t j − t i n y − r a n d o m " , e n a b l e _ a t t e n t i o n _ p r o b s = T r u e , r e n a m e _ c o n f i g = R e n a m e C o n f i g ( a t t n _ p r o b _ s o u r c e = G P T J A t t n P r o b F u n c t i o n ( ) ) ) 6 The process involves: (1) using model.scan() to explore the forward pass, (2) locating where attention probabilities are computed (typically after dropout), (3) implementing the hook via AttnProbFunction, and (4) testing with dummy inputs to verify shapes and normalization. A.4 Validation nnterp automatically validates configurations during initialization, checking: module naming cor- rectness, tensor shapes at each layer, attention probabilities normalization (if enabled), and I/O com- patibility. Manual validation can be performed using model.trace() to verify expected tensor shapes and properties. B Model coverage The following model classes were tested and work with nnterp: • BloomForCausalLM • BloomModel • Ernie4_5_MoeForCausalLM • GPT2LMHeadModel • GPTBigCodeForCausalLM • GPTJForCausalLM • Gemma2ForCausalLM • Gemma3ForCausalLM • Gemma3ForConditionalGeneration • GemmaForCausalLM • Glm4ForCausalLM • Glm4MoeForCausalLM • LlamaForCausalLM • MistralForCausalLM • MixtralForCausalLM • OPTForCausalLM • Phi3ForCausalLM • Qwen2ForCausalLM • Qwen3ForCausalLM • Qwen3MoeForCausalLM • SeedOssForCausalLM • SmolLM3ForCausalLM • DbrxForCausalLM • GptOssForCausalLM • Qwen2MoeForCausalLM • StableLmForCausalLM Support for attention probabilities is still a work in progress for the following model classes: • DbrxForCausalLM • GptOssForCausalLM • Qwen2MoeForCausalLM • StableLmForCausalLM 7

Instant research discovery