Paper deep dive
Generative Interpretability via Scalable Neuro-Symbolic Models
Xiaocong Yang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/20/2026, 3:20:49 AM
Summary
The paper argues that post-hoc interpretability is inadequate for safe AI deployment in agentic systems because it cannot intervene before outputs are generated. It proposes 'generative interpretability' as a new paradigm where models natively expose human-understandable, causally intervenable checkpoints during inference. Neuro-Symbolic Models are presented as a concrete instantiation of this concept, specifically Type-3 models that combine neural and symbolic components in a feedback loop to achieve scalable, auditable autonomy.
Entities (9)
Relation Signals (8)
Neuro-Symbolic Models → instantiates → Generative Interpretability
confidence 96% · propose Neuro-Symbolic Models as a concrete instantiation.
Generative Interpretability → contrastswith → Post-hoc Interpretability
confidence 95% · We therefore argue for a shift toward generative interpretability... comparison to other interpretability research paradigms
Generative Interpretability → requires → Human Understandability
confidence 93% · exposes intermediate checkpoints that are (i) semantically meaningful to humans natively
Generative Interpretability → requires → Causal Intervention
confidence 93% · exposes intermediate checkpoints that are... (ii) causally intervenable by construction.
Type-3 Neuro-Symbolic Models → isinstanceof → Neuro-Symbolic Models
confidence 92% · Using the categorization criteria proposed in [6], the Type-3 Neuro-Symbolic Models... best satisfy this requirement.
NLProlog → isinstanceof → Type-3 Neuro-Symbolic Models
confidence 90% · A representative Type-3 model is NLProlog
Post-hoc Interpretability → includes → Transformer Circuit Theory
confidence 88% · With the development of LLMs, the Transformer Circuit Theory... become one of the most cutting-edge techniques in post-hoc interpretability.
→ →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:As the use of Large Language Models moves from chatbots into agentic systems, where outputs become actions with irreversible consequences on reality, the existing paradigm on AI Interpretability research, post-hoc interpretability, is structurally inadequate for safe and trustworthy model deployment: it explains behavior after the fact but cannot audit or intervene in an inference computation before it commits to an output. We therefore argue for a shift toward \emph{generative interpretability}, an architectural property under which a model's inference pass natively exposes semantically meaningful checkpoints that are human-understandable and amenable to causal intervention. We show the merits of generative interpretability as comparison to other interpretability research paradigms, and propose Neuro-Symbolic Models as a concrete instantiation.
Tags
Links
- Source: https://arxiv.org/abs/2609.13529v1
- Canonical: https://arxiv.org/abs/2609.13529v1
Trouble viewing inline? Open PDF directly →
Full Text
37,929 characters extracted from source content.
Expand or collapse full text
Generative Interpretability via Scalable Neuro-Symbolic Models Xiaocong Yang AI Interpretability @ Illinois, University of Illinois Urbana-Champaign Urbana, IL, USA xy51@illinois.edu Abstract As the use of Large Language Models moves from chatbots into agentic systems, where outputs become actions with irreversible consequences on reality, the existing paradigm on AI Interpretabil- ity research, post-hoc interpretability, is structurally inadequate for safe and trustworthy model deployment: it explains behavior after the fact but cannot audit or intervene in an inference computation before it commits to an output. We therefore argue for a shift toward generative interpretability, an architectural property under which a model’s inference pass natively exposes semantically meaning- ful checkpoints that are human-understandable and amenable to causal intervention. We show the merits of generative interpretabil- ity as comparison to other interpretability research paradigms, and propose Neuro-Symbolic Models as a concrete instantiation. CCS Concepts • Computing methodologies→Knowledge representation and reasoning; Natural language processing; Artificial intel- ligence. Keywords AI Interpretability, Neuro-Symbolic Models, Large Language Mod- els, AI Safety, Generative Interpretability ACM Reference Format: Xiaocong Yang. 2026. Generative Interpretability via Scalable Neuro-Symbolic Models. In ACM AI Summit 2026 (AI Summit ’26), August 31-September 02, 2026, Atlanta, GA, USA. ACM, New York, NY, USA, 6 pages. https: //doi.org/10.1145/3806096.3844850 1 Introduction Recent years witnessed the rapid development of connectionist AI models, especially Large Language Models (LLMs) with unprece- dented real-world impacts. Although powerful, they are generally believed to lack interpretability on their internal working mecha- nisms and are therefore described as "black-box" models [22,34]. A direct consequence is that it is usually infeasible to understand, audit and intervene in undesirable model behaviors such as halluci- nation, sycophancy and harmful content generation in a scalable and systematic way. The negative consequences of such undesired model behaviors are dramatically amplified as the trend of deploy- ing LLMs in agentic systems, such as healthcare systems that help with diagnoses [44] and autonomous driving systems that control This work is licensed under a Creative Commons Attribution 4.0 International License. AI Summit ’26, Atlanta, GA, USA © 2026 Copyright held by the owner/author(s). ACM ISBN 979-8-4007-2638-5/2026/08 https://doi.org/10.1145/3806096.3844850 the actions of a running vehicle [48]. In those systems, agents in- teract with the real-world environments and therefore can have irreversible consequences that are much more severe than in a chatbot: as reported by Maslej et al. [25], documented AI safety incidents surged 56.4% from 2023 to 2024, with autonomous driv- ing systems involved in fatal crashes [33] and agentic AI systems causing financial losses [5]. Those issues have risen to the urgency at the national security level: the DARPA-NSF-CAISI AI Forge pro- gram [1] identifies AI Interpretability as one of the three priorities in AI research, highlighting the importance of interpretable AI in agentic settings for auditable autonomy and operational runtime control. In this work, we argue that a paradigm transition is necessary for interpretability research to meet this requirement. Most current work on explaining LLMs falls into the category of post-hoc inter- pretability [23], which relies on reverse-engineering tools external to the LLM. Such tools are confined to observation: their interpreta- tions are confounded by the tools’ own approximation errors, and the tools themselves are usually discarded at deployment rather than used to make the deployed model safer. As a result, post-hoc interpretability cannot reliably translate into safer behaviors from deployed LLMs. Instead, we propose the concept of generative inter- pretability as an alternative. The key claim from this concept is that, interpretability should be a property inherent to the model computations that allows effective audit and intervention at deployment, directly targeting the research thrusts and challenges proposed in the DARPA-NSF-CAISI AI Forge report. 2 Post-hoc Interpretability and Its Limitations Post-hoc interpretability refers to the property where the computa- tions and behaviors of an already-trained model can be attributed and grounded into human understandable terms [23]. Based on the explanandum, they can be summarized into two categories: anatomical interpretability and physiological interpretability. 2.1 Anatomical Interpretability Anatomical interpretability studies local structures in a model that function stably and consistently over different inputs, usually ac- companied by modularity and functional specialization. For exam- ple, the anatomy of multi-head self-attention layers (MHAs) and feed-forward networks (FFNs) in a Transformer [42] have been thoroughly studied before the emergence of LLMs. For example, the MHAs in a BERT model [11] are discovered to specialize in pro- cessing certain semantic or syntactic information in input [9]. The FFNs in a Transformer are usually interpreted as factual knowledge banks [10, 15], which can be edited to influence model outputs on certain facts [26]. arXiv:2609.13529v1 [cs.LG] 11 Sep 2026 AI Summit ’26, August 31-September 02, 2026, Atlanta, GA, USAXiaocong Yang 2.2 Physiological Interpretability Physiological interpretability recovers the end-to-end information flow of a specific inference pass executed by the model. The theoreti- cal basis can be traced back to causal mediation analysis [31], where the computation flow on a given input sample is characterized as a Directed Acyclic Graph (DAG) and interventions are applied to intermediate nodes to study the causal effect on model output. Early efforts in this category include probing [4] and naive saliency trac- ing [20,37,38]. With the development of LLMs, the Transformer Circuit Theory [13] and its follow-up work [8,12,21] are proposed and become one of the most cutting-edge techniques in post-hoc interpretability. Instead of performing naive, metric-based tracing, Transformer Circuit Theory targets reducing an inference pass by LLMs to combinations of simplified operations on semantically meaningful directions called features in model activation spaces. These work systematically study and answer three key questions centered around features: (1) how are features distributed in activa- tion spaces (quasi-orthogonally); (2) how to identify features and their meanings (via dictionary learning); and (3) how does a model use these meaningful features to generate its output (via circuit tracing with attribution graphs). 2.3 The Shared Fundamental Limitation The work from post-hoc interpretability gives us insights to better understand LLMs. However, all of them have a fundamental limita- tion: they can explain what a model does, but cannot change what it does, in a systematic and scalable way. On the one hand, post-hoc interpretability tools are far from as effective in inference- time intervention as they are in representation explaining. For example, Basu et al. [3]documented a large gap between a model’s internal representations and its outputs that steering interventions based on different post-hoc interpretability tools failed to reliably close. On the other hand, post-hoc interpretability tools suffer from severe scalability and/or efficiency problems that block them from being deployed outside of laboratory environment. Those powerful interpretability tools in Transformer Circuit Theory, such as Sparse Autoencoders and Cross-layer Transcoders, cannot be efficiently integrated into LLMs to be deployed, due to the extremely large feature dimensions used to encourage sparse and clean feature de- composition. Other lightweight tools, such as model editing [43] and activation patching [52], are limited to modifying a specific type of model behavior with the dependency on labeled data related to that behavior, and hence far from scalable and systematic. 3 Generative Interpretability The empirical failure of intervention with post-hoc interpretabil- ity tools directly motivates the necessity of a new interpretability paradigm that is tailored for effective audit. In this section, we articulate generative interpretability as the appropriate require- ment on models to procure inference-time audit and controllable intervention. 3.1 Definition We define generative interpretability as follows. Definition 1 (Generative Interpretability). An archi- tectural property of a model under which its inference pass 1 exposes intermediate checkpoints that are (i) semantically mean- ingful to humans natively and (i) causally intervenable by construction. Human understandability. Interpretability in AI is typically framed in terms of human social attribution and cognition [7,27]. By semantically meaningful to humans, we require that a checkpoint to be mapped to a human-readable concept. Audit and intervention ultimately reflect human decisions and values, and they can only identify undesired model behaviors when the intermediate states are expressed in terms they understand. This excludes interpreta- tions that are only transparent in terms of mathematical function- alities. For instance, an optimization view yields a mathematically interpretable Transformer [49], whose layers have well-defined roles such as token set compression and sparsification, yet it does not reveal any semantic meanings the model assigns during pro- cessing inputs. By native, we require the acquisition of meanings not to rely on external tools to recover. For example, the framework by Mueller et al. [28]separates the choice of checkpoints from the discovery of their meanings; whether the checkpoints are Atten- tion heads, neurons or activation subspaces, their meanings rely on post-hoc interpretability tools to search and assign, and therefore, it cannot avoid the key defects of post-hoc interpretability. Causal intervention. An interpretability framework is consid- ered faithful when reflecting the real reasoning process of a model processing inputs [14,17]. In particular, the framework should be able to be tested with “do-intervention” [30] and the counterfactual output should align with the interpretation consistently, as argued in [29]. This is important to achieve predictable model behaviors when imposing intervention, although this is usually not the case with post-hoc interpretability tools. The requirement on causal in- tervention directly implies that the checkpoints need to be variables the inference pass functionally depends on, where altering their values necessarily propagates through the downstream computa- tion and so do-interventions have well-defined counterfactuals by construction, instead of contingent and approximate causal effects as with post-hoc interpretability tools. 3.2 Two Axes of Interpretability The definition above commits to two requirements but leaves open how generative interpretability relates to existing interpretability paradigms. We organize this comparison along two axes capturing distinct dimensions of commitment: when interpretation is avail- able relative to inference (the temporal axis), and where along the inference path human-readable structures reside (the spatial axis). The temporal axis. We divide interpretability approaches into three categories based on when human-understandable states are available relative to model inference. Most traditional machine learning models, such as linear regression models, decision trees, 1 We use the term “inference pass” to describe a single computation trajectory that maps input to a single meaningful output unit. For example, in auto-regressive LLMs, it is the generation of a next token. We use “inference path" for the static computational pathway defined by the model architecture, along which an inference pass is executed for a given input. Generative Interpretability via Scalable Neuro-Symbolic ModelsAI Summit ’26, August 31-September 02, 2026, Atlanta, GA, USA and rule-based systems, are interpretable before inference. Their structure or static parameters are human-understandable in them- selves and fully determine their behavior on every input, so the inference pass generates no extra interpretable information beyond a naive application of already-readable patterns. Rudin[34]advo- cates this property, and their follow-up work shows the superior performance by a fully interpretable, domain-specific model [16]. Those models are usually described as white-box models. The afore- mentioned post-hoc interpretability methods recover interpretable states only after inference: the human-understandable states are not natively exposed by the model but reconstructed by external tools from completed inference passes. The third category, where interpretable states emerge during inference passes, has drawn less attention. A representative work is Concept Bottleneck Models (CBMs) [19] and its extension to LLMs [36], where the computa- tions from neural networks are grounded in semantic concepts from a pre-defined, human-understandable concept pool as the final component in inference paths. However, these models are usually limited to simple similarity matching in a specific domain. The spatial axis. Interpretability approaches can also be catego- rized by the spatial position of interpretable states in an inference path. As a baseline, the vanilla LLM and interpretations based on it (such as Chain-of-thought [46]) offer human-readable represen- tations in form of text tokens only at input/output boundaries. All post-hoc interpretability methods rely on external tools to construct interpretations that lie externally to the model. The CBM family provide an interpretable interior checkpoint layer that allows simple concept matching, and most white-box models are interpretable throughout the model by definition. The position of generative interpretability. Along both axes, generative interpretability occupies an intermediate position. Tem- porally, it resides during inference. A checkpoint’s schema (con- cept space) is usually pre-defined, while the values filling it are pro- duced by an opaque sub-computation (such as a neural network) as the input is processed to that checkpoint. The interpretable states, therefore, can neither be derived from the static model alone, nor relying on the full inference pass to be completed. Spatially, it re- quires multiple interior checkpoints for the purpose of effective causal intervention. Boundary readability exposes only inputs and outputs, and a single checkpoint captures only one cross-section of a multi-step computation within a model, which is insufficient to attribute undesired output (this will be discussed more in Section 3.3). 3.3 Generative Interpretability as the Minimal Sufficient Commitment for Auditable LLMs We argue that, for LLMs, generative interpretability is the minimal architectural commitment sufficient to make audit and intervention over the inference pass that is scalable in development and compu- tationally tractable at deployment. The commitment is prescriptive: a property of the models we should build on existing LLMs, not as they are. A partial order by auditable content. The axes induce partial orders on commitment strength. Spatially, a throughout-interpretable model is legible at any interior point, subsuming multiple-checkpoint readability, which subsumes a single checkpoint and then the input- output boundary. Temporally, a model legible before inference re- mains legible during it when given an input. Post-hoc interpretabil- ity lies outside this chain, as its readable structure is reconstructed externally rather than residing in the model, and therefore implies no commitment on the model architectures. Why not weaker. Auditing content generation before it goes to output rules out after-inference interpretability immediately, whose readable state is recovered only once the output—and any unsafe content—already exists. During inference is thus the weakest temporal position that still permits pre-generation audit. Spatially, two weaker commitments fall short. The input-output boundary (vanilla LLMs, CoT) exposes no intermediate checkpoints on the inference path. Note that any audit on CoT’s intermediate steps is based on tokens that are already generated via inference passes, so it can only reject an undesired generation rather than attribute to the computation that produces it and correct accordingly. More- over, the intermediate steps in a CoT are not faithfully reflecting the internal computation [2,35,40], making intervention on them unreliable [41]. A single interior checkpoint as in CBM audits one cross-section, and in the setting of LLMs, it cannot attribute an undesired output to the computation either, as empirically verified in [3]. From a theoretical perspective, as proposed by Poggio[32], any efficiently computable function, characterized by efficient Tur- ing computability, is provably compositionally sparse with a DAG structure. Therefore, to meet the requirement of causal intervention as in Definition 1, we need at least multiple checkpoints to model such structures. Why not stronger. One might demand a stronger, white-box- style commitment. However, such semantic throughout-readability harms the representational capacity of the model. A representative is the effort to achieve monosemanticity of neurons or directions in activation spaces, supporting clean mapping between semantic concepts and model structures. However, this is infeasible for LLMs, which must represent far more meaningful concepts than they have dimensions, packing them into near-orthogonal directions through superposition [12]. A푑-dimensional space admits exponentially many such near-orthogonal directions [18], and forcing every direc- tion to be monosemantic would abandon this packing and collapse the number of meaningful concepts from exponential to linear in the dimension. Generative interpretability, however, only commits interpretability at limited checkpoints, leaving most of the model unconstrained to represent knowledge efficiently. 4 Neuro-Symbolic Model as an Instance with Generative Interpretability In the development of AI history, Neuro-Symbolic Model [6] is a successful instance that exploits the benefits from both Connec- tionism and Symbolism AI paradigms. From the perspective of information compression, neural and symbolic models process raw input data in fundamentally different ways. Symbolic systems com- press by projecting onto a pre-defined schema vocabulary, and the representations are discrete and hard-bounded that generalize in an algebraic manner [24], naturally providing human-understandable states. Neural networks, however, compress by projecting onto AI Summit ’26, August 31-September 02, 2026, Atlanta, GA, USAXiaocong Yang learned patterns that emerge during training as continuous, dis- tributed representations that generalize based on interpolation. Before Inference During Inference After Inference Throughout White-box Models [34] — Multiple Interior Checkpoints — Neuro-Symbolic Models — Single Interior Checkpoint — CBM [19], CB- LLM [36] — I/O Bound- ary Only — Vanilla LLM (CoT) [46] — External — Post-hoc Interp. (SAEs [39], circuits [21], probing [4], etc.) Temporal (earlier) Spatial (more pervasive) Figure 1: Interpretability paradigms organized by two tempo- ral and spatial axes. Shaded cells admit audit and interven- tion before undesired content is generated. Neuro-symbolic models realize generative interpretability within this region while preserving distributed, high-capacity representations. 4.1 Symbolic Components as Checkpoints Unlike the vague patterns in neural networks, the representations in a symbolic system are human-understandable by construction. The causality requirement immediately selects among different neuro-symbolic architectures that meets Definition 1. Using the categorization criteria proposed in [6], the Type-3 Neuro-Symbolic Models, where the neural and symbolic components work in a cooperative way in a feedback loop, best satisfy this requirement. A representative Type-3 model is NLProlog [45], where an inference pass consists of composable proof steps represented by predicate symbols, and similarity scoring produced by text embeddings inside each step stays opaque. The proof steps thus form multiple interior checkpoints within a single inference pass, each auditable against the expected intermediate representation. 4.2 Challenges of Building Neuro-Symbolic Models with LLMs Neuro-Symbolic paradigm is a promising candidate to mitigate the black-box nature of LLMs given by the intrinsic generative interpretability. However, most existing Neuro-Symbolic Models are built on a small-sized neural network component and confined to a specific task domain, as there are several key challenges that block scaling up. Challenge 1: Construction of a universal, general-purpose symbolic system. In existing Neuro-Symbolic Models, it is feasible to build the symbolic components from scratch by human experts for a narrow, closed use case and domain, such as logic rules for math theorem proving. However, this is intractable for building a universal symbolic system covering as many domains as an LLM does for general purpose, such as the Cyc project [50] starting from 1980s but paused in recent years of “information explosion”. A promising alternative approach is to build upon enormous existing symbolic systems from all different domains and use data-driven approaches to align, merge and extend into a colossal one to match the scope of LLMs, as is being studied by us concurrently. Challenge 2: Integration of neural and symbolic compo- nents. Although the Type-3 Neuro-Symbolic Model mentioned earlier gives us a sketchy prototype of the fusion of neural and symbolic components, the existing instances from this type usu- ally simplify into a rule-based, fixed workflow template that only works on a specific task. It seems difficult to characterize a universal workflow pattern that guides the design of general Neuro-Symbolic Models. A compromising approach is to abstract the shared work- flow template for a task category and build Neuro-Symbolic Models for it, such as the Matching Neural Network for Information Re- trieval proposed by Zhai[51], while the shared symbolic structure of the workflow is usually oversimplified. Challenge 3: The boundary between neural and symbolic components for appropriate generalization. The aforemen- tioned difference of information compression echoes a classical topic of knowledge itself in philosophy. Some knowledge is gov- erned by a continuous structure of similarity. For example, whether two unseen words are near-synonyms is judged by graded similar- ity instead of by any finite rule. Wittgenstein[47]coined a term “family resemblance” to refer to such knowledge, and neural net- works are naturally preferred to be used for their representation. Other knowledge is governed by discrete, exact rules. For instance, the correctness of a hash or of formal syntax is rule-defined, and any minor departure is simply wrong rather than approximately right, whose properties can be well captured by symbolic mod- els. Characterizing knowledge precisely, therefore, is meaningful for determining the proper boundary of the neural and symbolic components in a Neuro-Symbolic Model. 5 Conclusion We argued for a paradigm shift on AI interpretability research from explaining trained models (observation) to building auditable mod- els (intervention). Generative interpretability formalizes this: an ar- chitecture exposes semantically meaningful, human-understandable, and causally intervenable checkpoints along its inference pass, and Neuro-Symbolic Models instantiate this paradigm. We want to emphasize that this shift is not optional in agentic AI settings, es- pecially in high-stake scenarios such as healthcare, legal system and autonomous driving where the attribution of obligations and responsibility is also valued besides average performance on large samples. The audits at checkpoints on the inference path, ideally, mark a distinct locus of responsibility, so an undesired outcome traces to the step that bears it rather than to the system as a whole. Additionally, this mission requires efforts beyond technical work, including designing mechanisms of checks and balances on the power of AI audit, and therefore requires sustained collaborations among universities, frontier AI labs and governments. Generative Interpretability via Scalable Neuro-Symbolic ModelsAI Summit ’26, August 31-September 02, 2026, Atlanta, GA, USA References [1]2026. AI Forge: A National Partnership for AI Innovation. Technical Report. Defense Advanced Research Projects Agency (DARPA). https://w.darpa.mil/ sites/default/files/attachment/2026-06/ai-forge-report.pdf Joint program with NSF and NIST/CAISI. Distribution Statement A: Approved for Public Release. [2]Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. 2026. Chain-of-Thought Reasoning In The Wild Is Not Always Faithful. arXiv:2503.08679 [cs.AI] https://arxiv.org/abs/2503.08679 [3]Sanjay Basu, Sadiq Y. Patel, Parth Sheth, Bhairavi Muralidharan, Namrata Ela- maran, Aakriti Kinra, John Morgan, and Rajaie Batniji. 2026. Interpretability without actionability: mechanistic methods cannot correct language model er- rors despite near-perfect internal representations. arXiv:2603.18353 [cs.AI] https://arxiv.org/abs/2603.18353 [4]Yonatan Belinkov. 2021. Probing Classifiers: Promises, Shortcomings, and Ad- vances. arXiv:2102.12452 [cs.CL] https://arxiv.org/abs/2102.12452 [5]Yoshua Bengio et al.2026. International AI Safety Report 2026. Technical Report. In- ternational AI Safety Report Independent Expert Panel. arXiv:2602.21012 [cs.AI] https://arxiv.org/abs/2602.21012 [6]Bikram Pratim Bhuyan, Amar Ramdane-Cherif, Ravi Tomar, and Thipendra P. Singh. 2024. Neuro-symbolic artificial intelligence: a survey. Neural Computing and Applications 36, 21 (2024), 12809–12844. doi:10.1007/s00521-024-09960-z [7] Or Biran and Courtenay V. Cotton. 2017. Explanation and Justification in Machine Learning : A Survey Or. https://api.semanticscholar.org/CorpusID:3911355 [8]Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. 2023. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread (2023). https://transformer- circuits.pub/2023/monosemantic-features/index.html. [9] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What Does BERT Look at? An Analysis of BERT’s Attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Tal Linzen, Grzegorz Chrupała, Yonatan Belinkov, and Dieuwke Hupkes (Eds.). Association for Computational Linguistics, Florence, Italy, 276–286. doi:10. 18653/v1/W19-4828 [10]Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge Neurons in Pretrained Transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 8493–8502. doi:10.18653/v1/2022.acl- long.581 [11]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR abs/1810.04805 (2018). arXiv:1810.04805 http://arxiv.org/abs/1810.04805 [12]Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022.Toy Models of Su- perposition. Transformer Circuits Thread (2022).https://transformer- circuits.pub/2022/toy_model/index.html. [13]Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Das- Sarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021. A Mathemat- ical Framework for Transformer Circuits. Transformer Circuits Thread (2021). https://transformer-circuits.pub/2021/framework/index.html. [14]Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. 2025. Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. arXiv:2301.04709 [cs.AI] https://arxiv.org/abs/2301. 04709 [15]Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 5484–5495. doi:10.18653/v1/2021.emnlp-main.446 [16]Xiyang Hu, Cynthia Rudin, and Margo Seltzer. 2023. Optimal Sparse Decision Trees. arXiv:1904.12847 [cs.LG] https://arxiv.org/abs/1904.12847 [17]Alon Jacovi and Yoav Goldberg. 2020. Towards Faithfully Interpretable NLP Sys- tems: How should we define and evaluate faithfulness? arXiv:2004.03685 [cs.CL] https://arxiv.org/abs/2004.03685 [18]William B. Johnson and Joram Lindenstrauss. 1984. Extensions of Lipschitz mappings into Hilbert space. Contemporary mathematics 26 (1984), 189–206. https://api.semanticscholar.org/CorpusID:117819162 [19]Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. 2020.Concept Bottleneck Models. arXiv:2007.04612 [cs.LG] https://arxiv.org/abs/2007.04612 [20]Heyi Li, Yunke Tian, Klaus Mueller, and Xin Chen. 2019. Beyond saliency: Understanding convolutional neural networks from saliency prediction on layer- wise relevance propagation. Image and Vision Computing 83-84 (March 2019), 70–86. doi:10.1016/j.imavis.2019.02.005 [21] Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum Mc- Dougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. 2025. On the Biology of a Large Language Model. Transformer Circuits Thread (2025). https://transformer- circuits.pub/2025/attribution-graphs/biology.html [22]Zachary C. Lipton. 2017.The Mythos of Model Interpretability. arXiv:1606.03490 [cs.LG] https://arxiv.org/abs/1606.03490 [23]Andreas Madsen, Siva Reddy, and Sarath Chandar. 2022. Post-hoc Interpretability for Neural NLP: A Survey. Comput. Surveys 55, 8 (Dec. 2022), 1–42. doi:10.1145/ 3546577 [24] Gary F. Marcus. 2001. The Algebraic Mind: Integrating Connectionism and Cognitive Science. MIT Press, Cambridge, MA. doi:10.7551/mitpress/1187.001.0001 [25]Nestor Maslej, Loredana Fattorini, Raymond Perrault, et al.2025. Artificial Intelligence Index Report 2025. Technical Report. Stanford University Human- Centered Artificial Intelligence (HAI). https://aiindex.stanford.edu/report/ [26]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023. Locating and Editing Factual Associations in GPT. arXiv:2202.05262 [cs.CL] https://arxiv. org/abs/2202.05262 [27] Tim Miller. 2017. Explanation in Artificial Intelligence: Insights from the Social Sciences. CoRR abs/1706.07269 (2017). arXiv:1706.07269 http://arxiv.org/abs/ 1706.07269 [28]Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. 2025. The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis. arXiv:2408.01416 [cs.LG] https://arxiv.org/abs/2408.01416 [29]Nirit Nussbaum-Hoffer, Nitay Calderon, Liat Ein-Dor, and Roi Reichart. 2026. LLM Explainability with Counterfactual Chains and Causal Graphs. arXiv:2606.05972 [cs.LG] https://arxiv.org/abs/2606.05972 [30] Judea Pearl. 2009. Causal Inference in Statistics: An Overview. Statistics Surveys 3 (01 2009), 96–146. doi:10.1214/09-S057 [31]Judea Pearl. 2013. Direct and Indirect Effects. arXiv:1301.2300 [cs.AI] https: //arxiv.org/abs/1301.2300 [32]Tomaso Poggio. 2025. On efficiently computable functions, deep networks and sparse compositionality. arXiv:2510.11942 [cs.LG] https://arxiv.org/abs/2510. 11942 [33]Responsible AI Collaborative. 2024.AI Incident Database.https:// incidentdatabase.ai Accessed 2026. [34]Cynthia Rudin. 2019.Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. arXiv:1811.10154 [stat.ML] https://arxiv.org/abs/1811.10154 [35]Daniel Scalena, Sara Candussio, Luca Bortolussi, Elisabetta Fersini, Malvina Nis- sim, and Gabriele Sarti. 2026. Beyond the Commitment Boundary: Probing Epiphe- nomenal Chain-of-Thought in Large Reasoning Models. arXiv:2606.13603 [cs.LG] https://arxiv.org/abs/2606.13603 [36]Chung-En Sun, Tuomas Oikarinen, Berk Ustun, and Tsui-Wei Weng. 2025. Concept Bottleneck Large Language Models. arXiv:2412.07992 [cs.CL] https: //arxiv.org/abs/2412.07992 [37]Mukund Sundararajan and Amir Najmi. 2020. The many Shapley values for model explanation. arXiv:1908.08474 [cs.AI] https://arxiv.org/abs/1908.08474 [38]Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks. arXiv:1703.01365 [cs.LG] https://arxiv.org/abs/1703.01365 [39]Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDi- armid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. 2024. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Trans- former Circuits Thread (2024). https://transformer-circuits.pub/2024/scaling- monosemanticity/index.html [40]Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Lan- guage Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. arXiv:2305.04388 [cs.CL] https://arxiv.org/abs/ 2305.04388 [41]Karthik Valmeekam, Vardhan Palod, Kaya Stechly, Atharva Gundawar, and Sub- barao Kambhampati. 2026. Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens. Transactions on Machine Learning Research AI Summit ’26, August 31-September 02, 2026, Atlanta, GA, USAXiaocong Yang (2026). https://openreview.net/forum?id=gDE7YcRC3F J2C Certification. [42]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention Is All You Need. arXiv:1706.03762 [cs.CL] https://arxiv.org/abs/1706.03762 [43]Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jun- dong Li. 2024. Knowledge Editing for Large Language Models: A Survey. arXiv:2310.16218 [cs.CL] https://arxiv.org/abs/2310.16218 [44]Xiaoye Wang, Nicole Xi Zhang, Hongyu He, Trang Nguyen, Kun-Hsing Yu, Hao Deng, Cynthia Brandt, Danielle S. Bitterman, Ling Pan, Ching-Yu Cheng, James Zou, and Dianbo Liu. 2025. Safety challenges of AI in medicine in the era of large language models. arXiv:2409.18968 [cs.CY] https://arxiv.org/abs/2409.18968 [45]Leon Weber, Pasquale Minervini, Jannes Münchmeyer, Ulf Leser, and Tim Rock- täschel. 2019. NLProlog: Reasoning with Weak Unification for Question Answer- ing in Natural Language. arXiv:1906.06187 [cs.CL] https://arxiv.org/abs/1906. 06187 [46]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https: //arxiv.org/abs/2201.11903 [47]Ludwig Wittgenstein. 1953. Philosophical Investigations. Wiley-Blackwell, New York, NY, USA. [48] Xiyang Wu, Souradip Chakraborty, Ruiqi Xian, Jing Liang, Tianrui Guan, Fuxiao Liu, Brian M. Sadler, Dinesh Manocha, and Amrit Singh Bedi. 2025. On the Vulnerability of LLM/VLM-Controlled Robotics. arXiv:2402.10340 [cs.RO] https: //arxiv.org/abs/2402.10340 [49]Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin D. Haeffele, and Yi Ma. 2023. White-Box Transformers via Sparse Rate Reduction. arXiv:2306.01129 [cs.LG] https://arxiv.org/abs/2306.01129 [50]Deniz Yuret. 1996. The binding roots of symbolic AI: a brief review of the Cyc project. (03 1996). [51] ChengXiang Zhai. 2024. Large Language Models and Future of Information Retrieval: Opportunities and Challenges. 481–490. doi:10.1145/3626772.3657848 [52]Fred Zhang and Neel Nanda. 2024. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. arXiv:2309.16042 [cs.LG] https: //arxiv.org/abs/2309.16042