Cracking the Circuits: Mechanistic Interpretability in Large Language Models

Abstract

Mechanistic interpretability aims to uncover how internal components of large language models (LLMs) contribute to their overall behaviour. While previous research has revealed interpretable circuits in small-scale transformers, the field still lacks a systematic and mathematically grounded framework for understanding, validating, and comparing interpretability techniques. In this paper, we introduce a unified formalism and taxonomy for mechanistic interpretability, defining core concepts such as modular circuits, representational superposition, and polysemantic neurons. We categorise interpretability methods based on granularity, causal influence, robustness to perturbation, and transferability across tasks. To demonstrate the utility of this framework, we analyse the GPT2-small model on the Indirect Object Identification (IOI) task using TransformerLens. Our findings reveal that attention heads 7.1 and 8.6 play a consistent and causal role in performing the name-mover function. Activation patching experiments show that restoring these components recovers over 90 % of performance lost due to input corruption, while ablating them results in a 27.3 % accuracy drop. These results provide empirical support for the existence of sparse, interpretable circuits that underpin model reasoning. Visualisations of patching curves and attention patterns further substantiate this behaviour. This study offers a principled methodology for mechanistic analysis in language models, bridging theoretical insights with empirical validation. The proposed framework supports the development of transparent, trustworthy, and controllable AI systems, with potential for extension to larger-scale models and safety-critical applications.

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/11/2026, 12:37:51 AM

MATCH (m:Model)-[:PERFORMS_TASK]->(t:Task) RETURN m.name, t.name

MATCH (c:Component)-[:CAUSALLY_INFLUENCES]->(f:Function) RETURN c.name, f.name

Full Text

825 characters extracted from source content.