Paper deep dive
Learning Concept Bottleneck Models from Mechanistic Explanations
Antonio De Santis, Schrasing Tong, Marco Brambilla, Lalana Kagal
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/13/2026, 12:32:41 AM
Summary
The paper introduces Mechanistic Concept Bottleneck Models (M-CBM), a pipeline that constructs interpretable concept bottlenecks directly from a black-box model's internal features using Sparse Autoencoders (SAEs) and Multimodal LLMs for concept naming and annotation. It addresses information leakage and concept quality issues in prior CBMs by introducing the Number of Contributing Concepts (NCC) metric for sparsity control.
Entities (5)
Relation Signals (3)
M-CBM → uses → Sparse Autoencoders
confidence 100% · These concepts are extracted via Sparse Autoencoders (SAEs)
M-CBM → utilizes → Multimodal LLM
confidence 100% · subsequently named and annotated on a selected subset of images using a Multimodal LLM.
NCC → controls → Information Leakage
confidence 90% · we also introduce the Number of Contributing Concepts (NCC), a decision-level sparsity metric that extends the recently proposed NEC metric.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Concept Bottleneck Models (CBMs) aim for ante-hoc interpretability by learning a bottleneck layer that predicts interpretable concepts before the decision. State-of-the-art approaches typically select which concepts to learn via human specification, open knowledge graphs, prompting an LLM, or using general CLIP concepts. However, concepts defined a-priori may not have sufficient predictive power for the task or even be learnable from the available data. As a result, these CBMs often significantly trail their black-box counterpart when controlling for information leakage. To address this, we introduce a novel CBM pipeline named Mechanistic CBM (M-CBM), which builds the bottleneck directly from a black-box model's own learned concepts. These concepts are extracted via Sparse Autoencoders (SAEs) and subsequently named and annotated on a selected subset of images using a Multimodal LLM. For fair comparison and leakage control, we also introduce the Number of Contributing Concepts (NCC), a decision-level sparsity metric that extends the recently proposed NEC metric. Across diverse datasets, we show that M-CBMs consistently surpass prior CBMs at matched sparsity, while improving concept predictions and providing concise explanations. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2603.07343v1
- Canonical: https://arxiv.org/abs/2603.07343v1
Trouble viewing inline? Open PDF directly →
Full Text
91,620 characters extracted from source content.
Expand or collapse full text
Published as a conference paper at ICLR 2026 LEARNINGCONCEPTBOTTLENECKMODELS FROM MECHANISTICEXPLANATIONS Antonio De Santis 1,2∗ Schrasing Tong 2 Marco Brambilla 1 Lalana Kagal 2 1 Politecnico di Milano 2 MIT CSAIL ABSTRACT Concept Bottleneck Models (CBMs) aim for ante-hoc interpretability by learning a bottleneck layer that predicts interpretable concepts before the decision. State- of-the-art approaches typically select which concepts to learn via human specifica- tion, open knowledge graphs, prompting an LLM, or using general CLIP concepts. However, concepts defined a-priori may not have sufficient predictive power for the task or even be learnable from the available data. As a result, these CBMs of- ten significantly trail their black-box counterpart when controlling for information leakage. To address this, we introduce a novel CBM pipeline named Mechanistic CBM (M-CBM), which builds the bottleneck directly from a black-box model’s own learned concepts. These concepts are extracted via Sparse Autoencoders (SAEs) and subsequently named and annotated on a selected subset of images using a Multimodal LLM. For fair comparison and leakage control, we also intro- duce the Number of Contributing Concepts (NCC), a decision-level sparsity met- ric that extends the recently proposed NEC metric. Across diverse datasets, we show that M-CBMs consistently surpass prior CBMs at matched sparsity, while improving concept predictions and providing concise explanations. Our code is available athttps://github.com/Antonio-Dee/M-CBM. 1INTRODUCTION As AI systems become increasingly complex and embedded in high-stakes applications such as healthcare, autonomous driving, and defense, there is a growing demand for models that not only perform well but are also transparent and interpretable. To obtain explanations for AI decisions, we can generally take two approaches: (i) utilize post-hoc methods that try to gain insights into how black-box models produce their outputs, or (i) develop inherently transparent models that can explain their decisions by design (i.e., ante-hoc explainability) (Xu et al., 2019). A promising ante- hoc approach to explainability is Concept Bottleneck Models (CBMs), which are trained to first predict an intermediate set of interpretable concepts and then use these concepts to predict the final output. Recent practice typically instantiates this concept set a-priori, either specified by human experts (Koh et al., 2020), based on knowledge graphs (Yuksekgonul et al., 2023), by prompting an LLM (Yang et al., 2023; Oikarinen et al., 2023; Srivastava et al., 2024), or using general concepts extracted from pre-trained vision-language models (Rao et al., 2024). However, concepts defined a-priori may not have sufficient predictive power for the target task or even be learnable from the available data. As a result, state-of-the-art CBMs substantially underperform their black-box coun- terpart when controlling for information leakage. Beyond performance, a further reason not to fix concepts a-priori is that modern ML systems often equal or exceed human expertise, creating an op- portunity to use interpretability to learn from machines. For example, Schut et al. (2025) extracted concepts learned by the chess engine AlphaZero (Silver et al., 2017) and were able to teach them to grandmasters. Furthermore, mechanistic interpretability has recently made significant progress in extracting concepts learned by black-box models, in particular via Sparse Autoencoders (SAEs) (Bricken et al., 2023). Motivated by this, we ask whether CBMs built directly from a model’s own learned concepts can serve as interpretable approximations of their black-box counterparts. Because these concepts originate in the backbone, we expect them to be easier to learn and to have better ∗ Work done while visiting at MIT CSAIL. Correspondence toantonio.desantis@polimi.it. 1 arXiv:2603.07343v1 [cs.LG] 7 Mar 2026 Published as a conference paper at ICLR 2026 predictive power. To test this, we develop a novel CBM pipeline, which we refer to as Mechanistic CBM (M-CBM), and compare it to state-of-the-art CBMs in both task accuracy and its ability to learn concepts, showing significant improvements. 2RELATEDWORK Concept-based Explanations.Early approaches for explainable AI typically rely on saliency (Selvaraju et al., 2017) or attribution maps (Sundararajan et al., 2017) that show which part of the input (e.g., regions or pixels of an image) contribute the most to a decision. By contrast, concept-based methods aim to provide explanations in terms of higher-level, human-understandable concepts (e.g., stripes for a zebra). A seminal contribution to the field was TCAV (Kim et al., 2018), a method that investigates a model’s sensitivity to a user-defined concept by collecting a set of exam- ple images representing that concept. Later, De Santis et al. (2025) extended TCAV with per-instance concept attributions and saliency maps indicating where the concept is recognized. However, both methods have practical limitations as they require users to manually collect concept examples. To address this, unsupervised approaches have also been proposed (Ghorbani et al., 2019; Zhang et al., 2021; Fel et al., 2023; Bianchi et al., 2024) to automatically discover influential concepts. These methods typically perform some form of clustering of a model’s activations to extract groups of semantically similar inputs or cropped patches that correspond to a concept. However, with this approach, achieving completeness (i.e., extracting a concept set sufficient to recover the model’s prediction) remains a nontrivial task (Yeh et al., 2020). Mechanistic Interpretability.Mechanistic interpretability (MI) aims to comprehensivelyreverse- engineerdeep networks by converting their neurons and weights into interpretable features and algorithms. A central challenge to this ispolysemanticity, i.e., neurons often respond to unrelated features, so they cannot be mapped one-to-one with concepts (Olah et al., 2020). This could allow networks to learn far more features than there are neurons, which is known as thesuperposition hypothesis (Elhage et al., 2022). Recently, Bricken et al. (2023) showed this can be addressed post- hoc by disentangling features via Sparse Autoencoders (SAEs) that learn a sparse, overcomplete dictionary of monosemantic features. Given their effectiveness in both language (Gao et al., 2025) and vision (Gorton, 2024; Thasarathan et al., 2025), we also adopt SAEs for concept extraction in our pipeline. Another emerging trend in MI isautomated interpretability, i.e., using LLMs to automate the generation of natural language descriptions for the functional role of network components. This was first applied to explain language model neurons (Bills et al., 2023), but then also proved effective to explain vision models (Rott Shaham et al., 2024). We also use a similar approach to assign names to concepts extracted via SAEs. MI has also made progress in dissecting models into interpretable circuits (e.g., identifying algorithmic sub-structures within deep networks) via masking or patching procedures (Conmy et al., 2023). Those circuit-level analyses are currently not being used in our pipeline, but integrating them could be a promising future work. Concept Bottleneck Models.Concept Bottleneck Models (CBMs) are self-explaining neural net- works that learn a set of intermediate human-understandable concepts to solve a task. The term was first introduced by Koh et al. (2020), who trained CBMs using datasets featuring concept annota- tions. Later, Yuksekgonul et al. (2023) relaxed this requirement with post-hoc CBMs that learn a Concept Bottleneck Layer (CBL) using Concept Activation Vectors (CAVs) (Kim et al., 2018), only requiring manual selection of representative examples for each concept. Furthermore, when using a CLIP (Radford et al., 2021) backbone, they could learn concepts directly from text sourced from the ConceptNet (Speer et al., 2017) knowledge graph. Yang et al. (2023) later showed benefits in gener- ating the concept set with LLMs. Oikarinen et al. (2023) extended this paradigm also to non-CLIP backbones using CLIP-Dissect (Oikarinen & Weng, 2023) to map concept embeddings in CLIP to any backbone. A known problem, however, that exists across all CBMs isinformation leakage, i.e., the fact that the CBL inadvertently encodes hidden class-relevant patterns beyond the concept semantics, which can be quickly learned by the final predictor to improve its accuracy (Havasi et al., 2022). This issue is quite serious, as Yan et al. (2023) even showed that replacing concepts with random words can achieve similar accuracy. Information leakage also results in unsatisfying expla- nations, in which the most important concepts contribute significantly less than the sum of all other concepts, making the model basically a black-box. Furthermore, while for classical CBMs, leakage can at least be quantified using metrics based on ground-truth concept labels (Havasi et al., 2022; 2 Published as a conference paper at ICLR 2026 Black-Box Backbone ③Dataset Annotation ①Concept Extraction Features EncoderDecoder SAE Neurons Reconstructed Features Sparse Autoencoder (SAE) ②Concept Naming Multimodal LLM Highly active examples Non-active examples Prompt What is this neuron looking for? Concept Name person wearing a helmet NoYesNo NoYesYes NoYesNo ④Concept Bottleneck Model Concept Labels Multimodal LLM Dataset Black-Box Backbone Concept Bottleneck Layer (CBL) Concepts Sparse Linear Layer Classes Multilabel BCE Loss Stripes Grass Snow Ski Church Zebra Multiclass CE Loss Do these images contain the concept "person wearing a helmet"? Prompt Helmet Figure 1: Overview of the M-CBM pipeline. (1) Given a trained black-box backbone, we extract its features and learn sparse, disentangled concept directions using a Sparse Autoencoder (SAE). (2) A Multimodal LLM is prompted with examples of highly activating and non-activating images to assign concept names to each SAE neuron. (3) The MLLM then annotates a subset of the dataset containing an equal split of active and non-active examples, indicating the presence or absence of each concept in selected images. (4) Using these concept annotations, we train a Concept Bottleneck Layer (CBL) and a sparse linear classifier to predict target classes from the learned concepts. Zarlenga et al., 2023), this is not trivial for CBMs that learn concepts automatically. To address this, Srivastava et al. (2024) introduced the Number of Effective Concepts (NEC) as a metric to measure and control how many concepts CBMs use to make a prediction, effectively reducing infor- mation leakage. In this work, we also follow this idea and use the Number of Contributing Concepts (NCC), a generalization of NEC, to control for leakage and explanation conciseness. More details on NEC and NCC are provided in Section 4. Srivastava et al. (2024) also introduced VLG-CBM, a CBM pipeline that uses GroundingDINO (Liu et al., 2025), an open-vocabulary object detector, to automatically annotate a dataset with LLM-generated concepts. The CBL is then trained on these annotations in a multilabel setting. However, leakage still arises from the annotation being class-conditioned, as we show in Section J. Another limitation of these CBM paradigms is that LLM-generated concept sets may not have sufficient predictive power for the target task or not even be learnable from the available data, requiring the inclusion of uninterpretable components in the bottleneck to recover accuracy (Yuksekgonul et al., 2023; Zhang et al., 2025; Shang et al., 2024). Sometimes concepts can also be non-visual (Roth et al., 2023) (e.g., “spicy”, “loud”), making ex- planations less transparent. Instead, we propose extracting and using the black-box model’s own learned concepts, rather than guessing with an LLM. A first step in this direction is DN-CBM (Rao et al., 2024), which learns concepts from CLIP with an SAE and uses its hidden layer as CBL, naming the concepts by selecting the nearest text embeddings to the decoder vector. However, this paradigm can only be applied with a CLIP backbone, limiting its accuracy across datasets, as we show in Section 6, and CLIP dependence can still introduce non-visual concepts. 3METHODOLOGY In this section, we introduce our methodology for transforming any black-box model into an interpretable-by-design CBM. Our approach, which we refer to as Mechanistic CBM (M-CBM), extracts human-interpretable concepts from a trained black-box model, assigns names and annota- tions using a Multimodal Large Language Model (MLLM), and then trains a sequential CBM (Koh et al., 2020) using these concepts. An overview of the whole pipeline is provided in Figure 1. 3 Published as a conference paper at ICLR 2026 Concept Extraction.Given a black-box backboneφtrained on an arbitrary datasetD, the first step of our methodology is to decompose the features learned by the model during training into a set of interpretable concepts. To achieve this, we use the Sparse Autoencoder (SAE) approach, which was recently popularized in the mechanistic interpretability literature (Bereska & Gavves, 2024) and has proven effective to disentangle model features into interpretable concepts for both vision (Gorton, 2024; Thasarathan et al., 2025) and language models (Bricken et al., 2023; Huben et al., 2024). An SAE is a neural network trained to reconstruct its input features while enforcing sparsity in the hidden representation (see step 1 ⃝in Figure 1). In our case, the input features are the activations a (i) =φ(x (i) )∈R n of the backboneφfor each samplex (i) in the training setD. Following Bricken et al. (2023), the SAE subtracts an input biasb D and then passes the resulting vector to an encoder with weightsW E , biasb E , and ReLU activation, obtaining the hidden layerh∈R m : h= ReLU W ⊤ E (a−b D ) +b E In the sparse hidden layerh, ideally, each neuron learns to recognize a distinct concept. A decoder with weightsW D and biasb D then mapshback to the reconstructed features ˆ a: ˆ a=W ⊤ D h+b D whereW E ∈R n×m ,W D ∈R m×n , and typically for large datasetsm≫nto account for the superposition hypothesis (Elhage et al., 2022), i.e., the fact that neural networks tend to learn more concepts than the neurons they have. The input and output biasesb D are opposite in sign and equal in magnitude. While Bricken et al. (2023) train SAEs with expansion factors (defined asm/n) ranging from1xto256x, we avoid going above4xto keep the annotation step computationally feasible. To train the SAE, we minimize the following objective: L SAE = 1 |D| |D| X i=1 ∥a (i) − ˆ a (i) ∥ 2 2 +λ SAE ∥h (i) ∥ 1 (1) whereλ SAE >0is a hyperparameter that controls the strength of the sparsity penalty on the hidden representation. We also monitor the averageℓ 0 norm (i.e., the number of non-zero activations) to ensureℓ 0 ≪n, as recommended by Bricken et al. (2023). SAE training often leaves many neurons in the hidden layerhdead (never activated for any training sample) or nearly dead (activate only very rarely). To ensure that our set of candidate concepts is both meaningful and computationally efficient for subsequent annotation, we perform a filtering step to remove such neurons. To define the threshold for identifying nearly dead neurons, we measure, for each unit inh, the number of training samples for which it is active. We then select a cutoff value such that removing all units below this threshold does not reduce the recovered cross-entropy loss of the black-box model, defined as1− L B ( ˆ a)−L B (a) L B (0)−L B (a) , by more than a tolerance of∼1%. This procedure ensures that only neurons with negligible contribution to predictive performance are pruned. This metric was also used to evaluate SAE quality in prior work (Bricken et al., 2023; Rajamanoharan et al., 2024; Gao et al., 2025). More details on SAEs in Appendix B. Concept Naming.After pruning, each remaining neuron in the SAE hidden layerhis treated as a candidate concept, where we denote byh j thej-th hidden SAE neuron. To assign human- interpretable names, we adopt an automated procedure inspired by recent work on mechanistic in- terpretability of language model neurons (Bills et al., 2023). For each candidate concept, we first select a set of inputsx∈Dthat maximally activate the corresponding neuronh j . For these in- puts, we also highlight the spatial regions that contribute the most to the activation, similarly to Rott Shaham et al. (2024). To compute these concept saliency maps, we use the method introduced by De Santis et al. (2025) (i.e., weighted average of feature maps usingW D as concept weights and followed by ReLU). To provide a contrastive signal, we additionally sample a set of non-activating examples, of which half are drawn at random fromD, and half are selected as the most cosine simi- lar to the activating examples to enhance discrimination of fine-grained visual features. The paired examples are provided to an MLLM, GPT-4.1 in our experiments, which is prompted to produce a concise natural-language description of the concept that the neuron represented byh j is responding to. At this stage, we also explicitly instruct the model not to use class names as concepts and re-try if it does not adhere. Step 2 ⃝of Figure 1 shows an example of what the MLLM receives as input. In our experiments, we used 10 activating examples and 10 non-activating ones. 4 Published as a conference paper at ICLR 2026 Finally, since we do not want duplicate or semantically equivalent concepts, we perform a merging step similar to Oikarinen et al. (2023), in which we embed all proposed textual names using a pre- trained embedding model and merge those with very high cosine similarity (i.e., above0.98). We use OpenAI’stext-embedding-3-largein our experiments. To make the embeddings context-aware, we also wrap each concept name in the following template before inserting it into the embedding model: “This is a visual concept in the context ofdomain:concept”. The variableconcept contains the concept name, whiledomainspecifies the dataset domain (e.g., bird species, skin lesions). For simplicity, Figure 1 omits the merging step. Dataset Annotation.With concept names assigned, we proceed to build a partially annotated dataset to train the Concept Bottleneck Layer (CBL). This step is necessary because a concept name is only a hypothesis, rather than a faithful description of the corresponding SAE neuron’s functional role. Since such hypotheses are often difficult to validate (Sharkey et al., 2025), we do not consider it ideal to use the SAE hidden layer directly as a bottleneck when interpretability is the primary goal. LetC=c 1 ,...,c K denote the final set of concepts. For each conceptc k , the goal is to obtain binary presence/absence labels on a subset of imagesx∈D. Since exhaustive annotation of the full dataset is not computationally feasible at the time of writing this paper, we annotate up to1000 samples per concept. The annotation procedure is performed by prompting the MLLM with batches of25images arranged in a5×5grid, which is much cheaper than one image at a time, but also slightly less accurate (see Appendix G). The model is asked to indicate, for each of the25grid images, whether the concept is present or absent. See step 3 ⃝of Figure 1 for a high-level overview of the annotation procedure. Each call also includes a grid of the top-25most activating images for the corresponding SAE neuron, which serve as a reference together with the textual concept name. To select the subset of images for annotation, we first select up to500active samples per concept. The active set is defined as all inputs for whichh j >0. From this set, we select samples whose activation lies above the 95th percentile of the set. If fewer than500samples exceed this percentile, we take the top-500activations overall within the active set. If the neuron has fewer than500active samples in total, we take all available examples, rounding the number down to the nearest multiple of25to match the batch annotation protocol. For merged neurons, activations are normalized across the group and treated as a single unit when computing percentiles. We then select an equal number of non-active samples, of which half are drawn uniformly at random and half are chosen as the most cosine similar to the active samples, similarly to the naming procedure. Furthermore, to avoid biasing concepts toward particular classes, both active and non-active sets are stratified across class labels. Each batch of 25 images also contains a balanced mix of active and non-active examples. At the end of this annotation step, we obtain a set of around1000annotated samples for each concept, containing both presence and absence cases across both training and test data. An image may be annotated for more than one concept or for none. Formally, for each imagex (i) ∈D, the annotation procedure creates a ternary vector of concept labelsz (i) ∈−1,0,1 K with the following entries: z (i) k = 1ifc k is annotated aspresentinx (i) 0ifc k is annotated asabsentinx (i) −1ifc k isnot annotatedforx (i) (2) Concept Bottleneck Model.After generating the concept labels, we proceed with training a se- quential CBM (Koh et al., 2020). As shown in step 4 ⃝of Figure 1, the CBM has three components: (i) a frozen backboneφthat maps an input image to a feature vector, (i) a Concept Bottleneck Layer (CBL)gthat predicts the presence ofKnamed concepts from those features in a multi-label setting, and (i) a sparse linear classifierfthat predicts the class from the concept outputs. For each inputx (i) the frozen backbone producesn-dimensional featuresa (i) =φ(x (i) )∈R n . The CBLg:R n →R K takes these features as input and outputs concept logits, then a sigmoid produces probabilities ˆ z (i) =σ(g(a (i) ))∈[0,1] K . From the annotation pipeline, each image carries a ternary concept vectorz (i) ∈−1,0,1 K indicating present (1), absent (0), or not annotated (−1). Since not every image-concept pair is labeled, we traingonly on the entries we know. LetΩ = (i,k) :z (i) k ∈ 0,1be the set of annotated pairs. The CBL is optimized to minimize a masked Binary Cross-Entropy (BCE) loss that averages overΩ: L CBL = 1 |Ω| X (i,k)∈Ω BCE ˆz (i) k , z (i) k (3) 5 Published as a conference paper at ICLR 2026 Entries withz (i) k =−1are effectively ignored in the loss computation. Therefore, images without any concept annotation (all entries−1) are not used to train the CBL. Furthermore, since positives are often rarer than negatives, we weight each concept in the BCE by the ratio of its class imbalance. To map concepts to classes, we follow prior work (Srivastava et al., 2024; Yuksekgonul et al., 2023; Oikarinen et al., 2023) and train a sparse linear classifier on concept logits (i.e., CBL’s pre-sigmoid outputs), optimized using the GLM–SAGA solver (Wong et al., 2021). Since GLM–SAGA assumes standardized input features, wez-normalize (zero mean and unit variance) the concept logits and use these to predict the classes. Withgfrozen, we define a fully connected layerf:R K →R C with weightsW F ∈R K×C and biasb F ∈R C , whereCis the number of output classes, and minimize the following Cross-Entropy (CE) loss with an elastic-net (Zou & Hastie, 2005) penalty: L CLF = 1 |D| |D| X i=1 CE f◦g◦φ(x (i) ),y (i) +λ CLF R α (4) wherey (i) represents the one-hot ground-truth class label for samplex (i) andR α = (1− α) 1 2 ∥W F ∥ 2 2 +α∥W F ∥ 1 denotes the elastic-net penalty. Following Wong et al. (2021), we use α= 0.99, whileλ CLF is tuned to obtain a target sparsity. 4NUMBER OFCONTRIBUTINGCONCEPTS(NCC) Prior work has shown that sparse layers are more interpretable (Wong et al., 2021; Yuksekgonul et al., 2023; Oikarinen et al., 2023), and Srivastava et al. (2024) also showed that sparsity is in- versely correlated with information leakage. They demonstrated that a dense linear classifier built on top of a random (i.e., untrained) CBL can recover black-box accuracy if the number of concepts Kapproaches or exceeds the backbone feature dimensionn, but this effect decreases with higher sparsity. Related studies (Shang et al., 2024; Yan et al., 2023) similarly report that when the concept set is large enough (e.g.,K≳n/2), dense linear classifiers can preserve black-box accuracy by re-estimating the backbone activations, therefore even using random words as concepts can match the accuracy obtained with concepts defined by LLMs or humans. While high sparsity improves interpretability and limits leakage, it naturally tends to correlate with lower accuracy (Wong et al., 2021; Oikarinen et al., 2023; Srivastava et al., 2024), making CBM comparison incomplete if only accuracy is reported. To address this, Srivastava et al. (2024) intro- duced an evaluation metric named NEC, which is defined as the average number (per-class) of non- zeros in the weightsW F of the final layerf. They trainfat different regularization strengthsλ CLF and accuracies are compared at equal NEC. This is convenient for enabling a fair comparison be- tween CBMs, but it also has limitations. Controlling NEC forces concise decision explanations, but it does so by linearly restricting the effective concept vocabulary as the number of classes decreases. For instance, with three classes, NEC=5forcesK≤15(or evenK= 5for binary classification with single output head) after training so that on average predictions are explained by∼5concepts. However, in datasets with substantial intra-class diversity (e.g., peeled or in-field pineapples are the same class in ImageNet), a class may require a rich concept vocabulary (i.e., largerK) to cover its different contexts, even though only a subset of them is needed to predict an individual image. With this in mind, we introduce a generalization of NEC, named Number of Contributing Concepts (NCC), which does not impose a hard cap onKbut still enforces concise explanations by measuring sparsity at decision-level using concept contributions rather than weights count. To measure the con- tribution of conceptk, for classrand imagei, we must consider the magnitude of both the concept logitg(a (i) )] k and its weight[W F ] k,r towards classr. We then define the absolute contribution of a concept to a class asu (i) k,r = [g(a (i) )] k ·[W F ] k,r . Ideally, we want the model to recognize a class with only a small subset of concepts that cover the vast majority of the total absolute contribu- tion, or, in other words, explain the vast majority of the decision. Letu (i) (s),r denote thes-th largest absolute contributing concept, and fix a coverage levelτ∈[0,1]. We define NCC as: NCC τ = 1 |D|C |D| X i=1 C X r=1 min n κ∈0,...,K: κ X s=1 u (i) (s),r ≥τ K X k=1 u (i) k,r o While here we average NCC over allCclasses, it can also be alternatively computed considering only the model’s predicted class. Intuitively, NCC τ is the average number of concepts required to 6 Published as a conference paper at ICLR 2026 explain at least aτfraction of the prediction of a class. For example, an NCC=5withτ= 0.95, means that, on average, just 5 concepts explain≥95%of the decision. For controlling NCC, we fix aτand follow the approach of Srivastava et al. (2024), trainingfat differentλ CLF and compare CBM accuracies at equal NCC levels. In practice, targeting a lower NCC generally means trading accuracy for explanation conciseness and vice versa. 5EXPERIMENTALSETUP Baselines.We compare M-CBM with three state-of-the-art CBMs: LF-CBM (Oikarinen et al., 2023), VLG-CBM (Srivastava et al., 2024), and DN-CBM (Rao et al., 2024). For VLG-CBM, we compare with a class-agnostic annotation variant, which we refer to as VLG-CBM CA , rather than the original pipeline. In the original VLG-CBM, concepts are assigned to classesbeforeannotation and are annotated only on images of their assigned classes. While this design reduces annotation cost, coupling concepts to classes can introduce substantial information leakage. We verify this on CUB in Figure 2. Using random words as concepts, VLG-CBM reaches black-box level accuracy already at around NCC=1.5, showing that in this setup, performance is insensitive to both sparsity and con- cept semantics. Intuitively, this happens because learning a concept that is labeled as positive only on images of a class is nearly equivalent to learning that class directly. When we remove class condi- tioning by annotating each concept across all images (VLG-CBM CA ), accuracy drops substantially for both random and real concepts, and the expected interpretability–accuracy trade-off reappears. We performed the same experiment with our M-CBM, and the performance of substituting concepts with random words is similar to using random words in VLG-CBM CA . However, when real con- cepts are used, our M-CBM outperforms VLG-CBM CA at high sparsity (NCC=3 to 5), while for low sparsity (NCC=10+), accuracy becomes similar to the random words variant due to information leakage. Further implementation details for this experiment are provided in Appendix C. (a) VLG-CBM(b) VLG-CBM CA (c) M-CBM (Ours) Figure 2: Accuracy vs NCC (τ= 0.95)on CUB. (a) With class-conditioned annotation, VLG-CBM reaches near black-box accuracy with NCC=1.5 (i.e., using only 1 to 2 concepts per prediction). The same happens using random concept names, showing evidence of leakage. (b) Making annotation class-agnostic (VLG-CBM CA ) restores the accuracy–interpretability trade-off, with real concepts slightly beating random words at low NCC. (c) M-CBM outperforms both VLG-CBM CA and the random baselines at low NCC, while leakage is significant for both methods as NCC increases. Setup.We evaluate on three standard image classification datasets that vary in domain and class count: CUB (Wah et al., 2011), ISIC2018 (Codella et al., 2019; Tschandl et al., 2018), and Im- ageNet (Deng et al., 2009). CUB contains∼6ktraining images and∼5.8ktest images of200 fine-grained bird species. As backbone for this dataset, we use the pre-trained ResNet18 frompy- torchcv. ISIC2018 contains dermatoscopic images of pigmented lesions categorized in 7 classes, split into∼10ktrain,193validation, and∼1.5ktest. Given a high class imbalance, we report both accuracy and balanced accuracy for this dataset. Given the lack of public pre-trained models, we train a ResNet50 (weighting each class by its imbalance ratio) and use it as a backbone. ImageNet includes1kclasses with∼1.3M training and50ktest images for general image classification. As backbone, we use the pre-trained ResNet50 fromtorchvision. Furthermore, for ImageNet and CUB, we extract10%from the train and use it as a validation set. Regarding DN-CBM, since it only supports a CLIP backbone, we evaluate it using both the ResNet50 and ViT-B/16 backbones. As discussed in Section 4, we compare under the same NCC. We useτ= 0.95and measure accuracies at NCC=5 and NCC=avg, with the latter being the average of the levels: 5, 10, 15, 20, 25, 30. 7 Published as a conference paper at ICLR 2026 Compute Resources.We trained all neural components (SAE, CBL, and GLM-SAGA) on an HPC cluster using an NVIDIA H200 on a multi-core node (32 cores and 512GB of RAM). On CUB and ISIC2018, each stage takes 5-20 minutes, while 5-10 hours for ImageNet. The dominant step in terms of cost and runtime is the annotation with GPT-4.1 API, which takes around 2 minutes and costs USD 0.14 per concept. Concept naming was lighter, taking around 10-20 seconds and USD 0.02 per concept, while concept merging costs were negligible. These costs scale linearly with the concept number, which was 278, 73, and 2648, respectively for CUB, ISIC2018, and ImageNet. 6RESULTS ANDDISCUSSION Accuracy Comparison.We report results in Table 1. Our M-CBM consistently achieves the high- est accuracy across datasets and NCC values. An expected interpretability-accuracy trade-off is also visible across all methods, as accuracy always increases when NCC is higher (i.e., explanations are less concise). DN-CBM consistently performs poorly, especially at NCC=5, indicating that a small subset of generic CLIP concepts may be insufficient to predict a class across datasets. VLG-CBM CA shows better accuracy than LF-CBM and DN-CBM, but annotating per-concept the entire dataset with GroundingDINO makes it computationally prohibitive at ImageNet scale (∼300GPU-days). In contrast, M-CBM uses SAE activations to pre-select candidate images per concept, so that we only need to annotate∼1kimages per concept. We exclude class-conditioned VLG-CBM from the comparison because, due to leakage, it is effectively a black-box (see Section J). Table 1: Accuracy comparison at NCC=5 and NCC=avg with best model in bold. The results for M-CBM are averaged over 3 seeds with same annotations. N/A denotes computationally unfeasible. DatasetCUBISIC2018ImageNet MetricsAccuracyAccuracyBalanced AccuracyAccuracy Black-box76.67%79.37%75.37%76.15% SparsityNCC=5NCC=avgNCC=5NCC=avgNCC=5NCC=avgNCC=5NCC=avg LF-CBM58.08%71.09%61.44%67.55%64.29%67.30%62.20%69.08% DN-CBM RN 38.21%48.98%35.38%54.61%39.85%52.85%46.71%57.24% DN-CBM ViT 48.12%66.19%43.92%56.08%42.47%53.56%60.23%69.98% VLG-CBM CA 69.12%72.25%64.55%72.61%64.63%70.80%N/AN/A M-CBM (Ours) 73.70% ±0.13% 74.18% ±0.06% 72.75% ±0.10% 75.51% ±0.08% 70.14% ±0.09% 71.54% ±0.05% 72.18% ±0.21% 73.64% ±0.15% Evaluating Concept Prediction.We assess how well each method can learn its own concepts by also annotating the test set. Because these labels are not ground truth, high scores do not guarantee that the model is learning the concepts as intended, but only that they are at least internally consis- tent and learnable. Especially for ISIC2018, we found that LLM-generated concept sets are often non-visual (e.g., “warm to the touch”) or not in the data (e.g., “medical report”). Since M-CBM uses concepts extracted from the backbone, we expect some benefits in the concept predictions, which is what we see in Table 2. Another factor that could contribute to the lower performance is the capa- bility of GroundingDINO to annotate correctly, which may be inferior to asking GPT-4.1, especially for medical images. However, due to a lack of ground truth, this remains challenging to quantify. Table 2: ROC-AUC evaluation of concept predictions on test set. Each method is evaluated on its own concepts. We report the macro-average across concepts and the average of the worst 10%. DatasetCUBISIC2018ImageNet Metrics ROC-AUCROC-AUCROC-AUC MacroWorst-10%MacroWorst-10%MacroWorst-10% VLG-CBM CA 62.03%45.60%73.37%52.92%N/AN/A M-CBM (Ours)90.04% 79.05%80.57% 66.98%88.90% 78.36% 8 Published as a conference paper at ICLR 2026 (a) ImageNet(b) ISIC 2018 Figure 3: Sankey plots of concept–class weights of our M-CBM at NCC=5. Concepts on the left and classes on the right. Concepts with negative weights are labeled as “NOT concept”. Explanations.We illustrate the behavior of M-CBMs through global (class-level) and local (instance-level) explanations, using models at NCC=5. Using the final layer weightsW F , we can visualize how concepts globally contribute to classes. In Figure 3, we show these weights using Sankey diagrams, with “NOT concept” indicating a negative weight. For clarity, we include only concepts with|W F |>0.1. On ImageNet, the model’s behavior aligns with intuition. The classes “Modem” and “Radio” share concepts related to ports/switches and antennas, while they are mainly differentiated by the presence of indicator lights for class “Modem” versus control knobs for class “Radio”. On ISIC2018, the model learns a richer concept set for “Melanocytic nevus” than for “Der- matofibroma”, which could be explained by the large class imbalance. Still, the few concepts learned for “Dermatofibroma” seem reasonable considering dermatological literature (Zaballos et al., 2006). Some minor concepts for “Melanocytic nevus”, such as skin-tone–related terms, are less clear. This likely arises from the concept naming (step 2 ⃝), where visually highlighting the concept (in this case, the skin around the nevus) in the image can introduce mild artifacts that GPT-4.1 over-interpreted. CBMs can also explain individual predictions by showing, for an inputx (i) , the contribution of con- cepts to a classr. This contribution is computed directly by multiplying the logit of thek-th concept g(a (i) )] k with its corresponding weight[W F ] k,r towards classr. Concepts with a negative logit are indicated as “NOT concept”. We show two examples in Figure 4, including a correct CUB predic- tion and a misclassification on ISIC, where the model incorrectly sees “clustered blue-gray ovoid nests”, leading to a “Basal Cell Carcinoma” prediction. Zeroing this concept flips the decision to the correct class. In both cases, we see that the decision is largely explained by the top 4-5 concepts. More examples of explanations are provided in Appendix F and J. 1. clustered blue-gray ovoid nests (+3.32) 2. central dark blotch (-0.17) 3. pink or light red papule/nodule with a central depression or ulceration (+0.15) 4. NOT pink lesion with multiple red and brown ulcerated or crusted areas (-0.13) 5. Sum of all 68 other concepts (+0.03) 1. yellow bird with black facial mask around eyes (+4.8) 2. black mask around yellow face (+4.34) 3. bright yellow plumage with black cap (+1.28) 4. yellow upperpart with black body and beak (+0.38) 5. olive-green upperparts (+0.12) 6. Sum of all 273 other concepts (+0.05) (a) Correctly predictedHooded Warbler 1. clustered blue-gray ovoid nests (+3.32) 2. central dark blotch (-0.17) 3. pink or light red papule/nodule with a central depression or ulceration (+0.15) 4. NOT pink lesion with multiple red and brown ulcerated or crusted areas (-0.13) 5. Sum of all 68 other concepts (+0.03) 1. yellow bird with black facial mask around eyes (+4.8) 2. black mask around yellow face (+4.34) 3. bright yellow plumage with black cap (+1.28) 4. yellow upperpart with black body and beak (+0.38) 5. olive-green upperparts (+0.12) 6. Sum of all 273 other concepts (+0.05) (b)Melanocytic Nevuswrongly predicted asBCC Figure 4: Per-image explanations of M-CBM at NCC=5 for a correct prediction in CUB (a) and a misclassification in ISIC 2018 (b). Concepts with negative logit are labeled as “NOT concept”. 7CONCLUSION ANDLIMITATIONS We presented Mechanistic Concept Bottleneck Models (M-CBMs), a novel paradigm for training CBMs using concepts learned directly from a black-box backbone and automatically annotated by an MLLM. With this approach, we substantially improve over the state-of-the-art, both in terms of 9 Published as a conference paper at ICLR 2026 task accuracy and concept predictions. We are also able to keep explanations concise by control- ling final layer sparsity to achieve a target Number of Contributing Concepts (NCC). One limitation general to all CBMs is that we still lack a systematic way to assess whether concepts are learned as intended and not via spurious correlations. This is because the final layer is interpretable, but the concept prediction remains a black-box. Another limitation is that, while NCC allows us to con- trol the accuracy–leakage trade-off, it is still not enough to eliminate leakage, as CBMs trained on random words still achieve much higher accuracy than we would expect from random chance. In future work, it may be interesting to investigate whether adding more bottleneck layers can mitigate this by making it harder for information to leak through. Furthermore, compared to other base- lines, M-CBM is less plug-and-play, requiring some supervision to ensure that concepts extracted via SAE are interpretable (see Appendix B), and that the MLLM is providing high-quality annota- tions. Finally, the high computational cost of using MLLMs for annotations can also be considered a limitation, especially for large datasets. Despite these limitations, given that, due to computational constraints, we annotate only a small subset of images, there might be great potential for improve- ment with the advancements of MLLMs in both performance and efficiency. ACKNOWLEDGMENTS Antonio De Santis was supported by the Progetto Rocca Doctoral Fellowship for his visit to MIT CSAIL. His doctoral scholarship is funded by the Italian Ministry of University and Research (MUR) under the National Recovery and Resilience Plan (NRRP), by Thales Alenia Space, and by the European Union (EU) under the NextGenerationEU project. REFERENCES Leonard Bereska and Stratis Gavves. Mechanistic interpretability for AI safety - a review.Transac- tions on Machine Learning Research, 2024. ISSN 2835-8856. URLhttps://openreview. net/forum?id=ePUVetPKu6. Survey Certification, Expert Certification. Matteo Bianchi, Antonio De Santis, Andrea Tocchetti, and Marco Brambilla. Interpretable net- work visualizations: a human-in-the-loop approach for post-hoc explainability of cnn-based im- age classification. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI ’24, 2024. ISBN 978-1-956792-04-1. doi: 10.24963/ijcai.2024/411. URL https://doi.org/10.24963/ijcai.2024/411. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders.Language models can explain neurons in language models.https://openaipublic.blob.core.windows.net/ neuron-explainer/paper/index.html, 2023. Trenton Bricken, Adly Templeton, Ben Chen, Jack Lindsey, Adam Jermyn, Shan Carter, Tom Henighan, Adam Pearce, and Chris Olah.Towards monosemanticity: Decomposing lan- guage models with dictionary learning.https://transformer-circuits.pub/2023/ monosemantic-features, 2023. Transformer Circuits Thread, Anthropic. Noel Codella, Veronica Rotemberg, Philipp Tschandl, M. Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic), 2019. URLhttps://arxiv.org/abs/ 1902.03368. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri ` a Garriga-Alonso.Towards automated circuit discovery for mechanistic interpretability.In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URLhttps: //openreview.net/forum?id=89ia77nZ8u. Antonio De Santis, Riccardo Campi, Matteo Bianchi, and Marco Brambilla. Visual-tcav: Concept- based attribution and saliency maps for post-hoc explainability in image classification, 2025. URL https://arxiv.org/abs/2411.05698. 10 Published as a conference paper at ICLR 2026 Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In2009 IEEE Conference on Computer Vision and Pattern Recognition, p. 248–255, 2009. doi: 10.1109/CVPR.2009.5206848. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superpo- sition, 2022. URLhttps://arxiv.org/abs/2209.10652. Thomas Fel, Agustin Picard, Louis B ́ ethune, Thibaut Boissin, David Vigouroux, Julien Colin, R ́ emi Cad ` ene, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 2711–2721, June 2023. Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. InThe Thirteenth Inter- national Conference on Learning Representations, 2025. URLhttps://openreview.net/ forum?id=tcsZt9ZNKD. Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch ́ e-Buc, E. Fox, and R. Gar- nett (eds.),Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. Liv Gorton. The missing curve detectors of inceptionv1: Applying sparse autoencoders to in- ceptionv1 early vision. InICML 2024 Workshop on Mechanistic Interpretability, 2024. URL https://openreview.net/forum?id=IGnoozsfj1. Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.),Advances in Neural Information Processing Systems, volume 35, p. 23386–23397. Curran Associates, Inc., 2022.URLhttps://proceedings.neurips.c/paper_files/paper/2022/ file/944ecf65a46feb578a43abfd5cddd960-Paper-Conference.pdf. Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum? id=F76bwRSLeK. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer Dy and Andreas Krause (eds.),Proceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, p. 2668–2677. PMLR, 10–15 Jul 2018. URLhttps://proceedings.mlr.press/v80/ kim18d.html. Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daum ́ e I and Aarti Singh (eds.),Proceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, p. 5338–5348. PMLR, 13–18 Jul 2020. URLhttps://proceedings. mlr.press/v119/koh20a.html. Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In Ale ˇ s Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and G ̈ ul Varol (eds.),Computer Vision – ECCV 2024, p. 38–55, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-72970-6. Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron representa- tions in deep vision networks.International Conference on Learning Representations, 2023. 11 Published as a conference paper at ICLR 2026 Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottle- neck models. InThe Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=FlCg47MNvBA. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits.Distill, 2020.doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.),Proceedings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, p. 8748–8763. PMLR, 18–24 Jul 2021.URLhttps://proceedings.mlr.press/v139/radford21a. html. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, J ́ anos Kram ́ ar, Rohin Shah, and Neel Nanda.Improving sparse decomposition of lan- guage model activations with gated sparse autoencoders.In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.),Advances in Neu- ral Information Processing Systems, volume 37, p. 775–818. Curran Associates, Inc., 2024.URLhttps://proceedings.neurips.c/paper_files/paper/2024/ file/01772a8b0420baec00c4d59fe2fbace6-Paper-Conference.pdf. Sukrut Rao, Sweta Mahajan, Moritz B ̈ ohle, and Bernt Schiele. Discover-then-name: Task-agnostic concept bottlenecks via automated concept discovery. InEuropean Conference on Computer Vision, 2024. Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 15700– 15711, 2023. doi: 10.1109/ICCV51070.2023.01443. Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, and Antonio Torralba. A multimodal automated interpretability agent. InForty-first International Conference on Machine Learning, 2024. Lisa Schut, Nenad Toma ˇ sev, Thomas McGrath, Demis Hassabis, Ulrich Paquet, and Been Kim. Bridging the human–ai knowledge gap through concept discovery and transfer in alp- hazero.Proceedings of the National Academy of Sciences, 122(13):e2406675122, 2025. doi: 10.1073/pnas.2406675122. URLhttps://w.pnas.org/doi/abs/10.1073/pnas. 2406675122. Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In2017 IEEE International Conference on Computer Vision (ICCV). IEEE, October 2017. doi: 10.1109/iccv.2017.74. URLhttp://dx.doi.org/10.1109/ICCV.2017.74. Chenming Shang, Shiji Zhou, Hengyuan Zhang, Xinzhe Ni, Yujiu Yang, and Yuwang Wang. In- cremental Residual Concept Bottleneck Models . In2024 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), p. 11030–11040, Los Alamitos, CA, USA, June 2024. IEEE Computer Society. doi: 10.1109/CVPR52733.2024.01049. URLhttps: //doi.ieeecomputersociety.org/10.1109/CVPR52733.2024.01049. Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeffrey Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Isaac Bloom, Stella Biderman, Adri ` a Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Mary Rumbelow, Martin Watten- berg, Nandi Schoots, Joseph Miller, William Saunders, Eric J Michaud, Stephen Casper, Max Tegmark, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Thomas McGrath. Open problems in mechanistic interpretability.Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URLhttps://openreview.net/forum? id=91H76m9Z94. Survey Certification. 12 Published as a conference paper at ICLR 2026 David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforce- ment learning algorithm, 2017. URLhttps://arxiv.org/abs/1712.01815. Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge.Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017. doi: 10.1609/aaai.v31i1.11164. URLhttps://ojs.aaai.org/index.php/ AAAI/article/view/11164. Divyansh Srivastava, Ge Yan, and Tsui-Wei Weng.Vlg-cbm:Training concept bottle- neck models with vision-language guidance.In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.),Advances in Neural In- formation Processing Systems, volume 37, p. 79057–79094. Curran Associates, Inc., 2024.URLhttps://proceedings.neurips.c/paper_files/paper/2024/ file/90043ebd68500f9efe84fedf860a64f3-Paper-Conference.pdf. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, p. 3319–3328. JMLR.org, 2017. Harrish Thasarathan, Julian Forsyth, Thomas Fel, Matthew Kowal, and Konstantinos G. Derpanis. Universal sparse autoencoders: Interpretable cross-model concept alignment. InForty-second International Conference on Machine Learning, 2025. URLhttps://openreview.net/ forum?id=UoaxRN88oR. Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.The ham10000 dataset, a large col- lection of multi-source dermatoscopic images of common pigmented skin lesions.Scientific Data, 5:180161, 2018. doi: 10.1038/sdata.2018.161. URLhttps://w.nature.com/ articles/sdata2018161. C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debug- gable deep networks. In Marina Meila and Tong Zhang (eds.),Proceedings of the 38th Inter- national Conference on Machine Learning, volume 139 ofProceedings of Machine Learning Research, p. 11205–11216. PMLR, 18–24 Jul 2021. URLhttps://proceedings.mlr. press/v139/wong21b.html. Feiyu Xu, Hans Uszkoreit, Yangzhou Du, Wei Fan, Dongyan Zhao, and Jun Zhu.Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges, p. 563–574. 09 2019. ISBN 978-3-030-32235-9. doi: 10.1007/978-3-030-32236-6 51. An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning Concise and Descriptive Attributes for Visual Recognition . In2023 IEEE/CVF International Conference on Computer Vision (ICCV), p. 3067–3077, Los Alamitos, CA, USA, October 2023. IEEE Computer Society. doi: 10.1109/ICCV51070.2023. 00287.URLhttps://doi.ieeecomputersociety.org/10.1109/ICCV51070. 2023.00287. Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable im- age classification. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 19187–19197, 2023. Chih-Kuan Yeh, Been Kim, Sercan ̈ O. Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. InProceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. 13 Published as a conference paper at ICLR 2026 Mert Yuksekgonul, Maggie Wang, and James Zou.Post-hoc concept bottleneck models.In The Eleventh International Conference on Learning Representations, 2023.URLhttps: //openreview.net/forum?id=nA5AZ8CEyow. P. Zaballos, ́ A. Llambrich, M. Ara, Z. Olazar ́ an, J. Malvehy, and S. Puig. Dermoscopic findings of haemosiderotic and aneurysmal dermatofibroma: report of six patients.British Journal of Der- matology, 154(2):244–250, 02 2006. ISSN 0007-0963. doi: 10.1111/j.1365-2133.2005.06844.x. URLhttps://doi.org/10.1111/j.1365-2133.2005.06844.x. Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. Towards robust metrics for concept representation evaluation. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’23/IAAI’23/EAAI’23. AAAI Press, 2023. ISBN 978-1-57735-880-0. doi: 10.1609/aaai.v37i10.26392. URLhttps://doi.org/10. 1609/aaai.v37i10.26392. Rui Zhang, Xingbo Du, Junchi Yan, and Shihua Zhang. The decoupling concept bottleneck model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(2):1250–1265, 2025. doi: 10.1109/TPAMI.2024.3489597. Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A. Ehinger, and Benjamin I. P. Rubinstein. Invertible concept-based explanations for cnn models with non-negative concept activation vec- tors.Proceedings of the AAAI Conference on Artificial Intelligence, 35(13):11682–11690, May 2021.doi: 10.1609/aaai.v35i13.17389.URLhttps://ojs.aaai.org/index.php/ AAAI/article/view/17389. Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net.Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301–320, 03 2005. ISSN 1369-7412. doi: 10.1111/j.1467-9868.2005.00503.x. URLhttps://doi.org/10.1111/ j.1467-9868.2005.00503.x. AAPPENDIXOVERVIEW In the appendix, we provide: B. Details on training SAEs C. Details on random words CBMs D. Summary of CBMs parameters and backbones E. Visualizations of CBL neurons F. More examples of explanations G. Additional experiments on dataset annotation H. Accuracies under NEC I. Visualizing SAE features J. Explanations compared with baselines K. Performance with an open-source MLLM BDETAILS ON TRAININGSAES In this section, we detail how we trained and evaluated SAEs for concept extraction. We did not find a single hyperparameter configuration that worked uniformly across datasets or backbones. Some dataset-specific adjustments were typically required. Following Bricken et al. (2023), we relied on a mix of quantitative and qualitative proxies to judge whether an SAE was “good enough” for downstream concept use. Specifically, we tracked the following metrics: 14 Published as a conference paper at ICLR 2026 1.L2 reconstruction loss. We want the reconstruction loss to be low to ensure we extract a comprehensive set of concepts. 2.Averageℓ 0 . We aim for a number significantly lower than the backbone dimensionality to ensure concepts are disentangled. 3.Feature density histogram. It shows how many hidden neurons fire at different activation frequencies across the training set. In the ideal scenario, neurons are either dead or rep- resent interpretable concepts, so we look for a histogram with two clusters, one with very low density representing dead or noisy features and one with higher density, which should represent the actual concepts. 4.Recovered cross-entropy loss. We ideally want the extracted concepts to recover model performance, so we know they have predictive power. 5.Recovered accuracy. Same as 4. For ISIC2018, we also consider balanced accuracy. 6.Manual inspection. Inspecting top-activating images for random neurons in the high- density cluster to assess whether the learned concepts seem interpretable. Empirically, we found that when all other metrics are healthy, most concepts are interpretable, although this cannot be guaranteed. In Table 3, we provide the training hyperparameters for the SAEs used in the paper, while in Table 4, we show the results in terms of the evaluation metrics we monitored. In Figure 5, we provide the Feature Density Histograms for each SAE. We can see that low-density neurons are generally well separated from high-density neurons. Furthermore, most of the low-density neurons are dead, i.e., never activating. Some neurons are neither dead nor high-density, and these are typically noisy and not very important for the task. As explained in Section 3, we perform a filtering step to remove these neurons before naming and annotation. In Figure 6, we show how choosing a different feature density cut-off impacts recovered loss, accuracy, and the number of neurons kept. We highlight the cut-off we used with a red star symbol. As we see, removing neurons with very low density has little impact on cross-entropy loss and accuracy. After pruning, recovered loss and accuracy for CUB become 89.40% and 98.41%. For ISIC2018, recovered loss and balanced accuracy become 99.41% and 96.84%. For ImageNet, recovered loss and accuracy become 97.63% and 96.60%. Table 3: Training hyperparameters for the SAEs used in the paper. HyperparameterCUBISIC2018ImageNet Backbone layer dimension51220482048 Expansion factor1×0.25×4× OptimizerAdamAdamAdam Learning rate1×10 −4 1×10 −4 1×10 −3 L1 coefficient (λ SAE )2×10 −3 5×10 −4 1×10 −3 Epochs100010001000 Patience for early stopping505050 Table 4: Evaluation metrics for the SAEs used in the concept extraction phase (pre-pruning). These are computed on the validation set, except forℓ 0 , which is computed on the training set. MetricCUBISIC2018ImageNet L2 reconstruction loss0.02310.00660.0462 Averageℓ 0 7.6617.1439.23 Recovered loss (CE)89.49%99.58%97.74% Recovered accuracy98.39%acc: 96.08%, bal. acc: 96.84%96.68% 15 Published as a conference paper at ICLR 2026 (a) CUB(b) ISIC 2018(c) ImageNet Figure 5: Feature density histogram for CUB, ISIC2018 and ImageNet. Purple indicates dead neu- rons (never active in the training set). Yellow indicates neurons that were pruned due to low density and little to no impact on recovered loss. Green indicates neurons that are kept as concepts for the subsequent steps. (a) CUB (b) ISIC 2018 (c) ImageNet Figure 6: Effect of pruning by activation-count threshold for CUB, ISIC2018, and ImageNet. Higher thresholds typically reduce recovered performance, but when discarding low-density neurons, the reduction tends to be negligible. The point highlighted by a red star indicates the cutoff used in our experiments. CDETAILS ON RANDOM WORDSCBMS In this section, we provide additional details on how we implemented the experiments with random words for VLG-CBM, VLG-CBM CA , and M-CBM. For each method, we replace every concept 16 Published as a conference paper at ICLR 2026 Figure 7: Examples of VLG-CBM explanations where replacing all concept names with random ones still yields correct predictions using just 1–2 concepts, illustrating how class-conditioned an- notation can leak class-specific information unrelated to concept semantics. name with a random, semantically meaningless text while preserving the cardinality of the original concept sets and their class-conditioned assignment for vanilla VLG-CBM. We draw words from the NLTK’s words corpus, filtered to lowercase alphabetic strings of length 3–8, and added the prefix “bird” so that the result is a short phrase like “bird pizza”. The prefix helps maintain min- imum image relevance so that models like GroundingDINO or GPT4.1 are more likely to annotate the random concepts as positive in some of the images. For VLG-CBM, the annotation is done using their official codebase without modifications, while for VLG-CBM CA , we remove class conditioning and annotate each (random) concept across all images. Because GroundingDINO accepts at most 256 input tokens, we batch concept lists and run multiple passes until all concepts are processed. For M-CBM, we follow our standard pipeline but substitute random names before annotation. Further- more, when annotating random concepts, we omit reference grids of top-activating images to avoid leaking information about the original concepts. Figure 7 illustrates how, under class-conditioned annotation, VLG-CBM can effortlessly predict correctly using only 1–2 random concepts. DSUMMARY OFCBMS PARAMETERS AND BACKBONES In this section, we report the number of parameters used at inference time for all methods and datasets. For each configuration, we decompose the total parameter count into (i) the pre-trained backbone, and (i) the CBL plus the final classifier, which we denote as CBM. The CBL sizeK corresponds to the dimensionality of the concept bottleneck (i.e., the total number of concepts). It differs across methods, because each CBM constructs and filters its concept set using its own procedure. Table 5 summarizes the parameter counts (in millions) for all methods considered in our main experiments. Differences are largely dominated by the choice of backbone (e.g., CLIP ViT- B/16 vs ResNet-18/50), while the additional parameters introduced by the CBM head are relatively small in all cases. 17 Published as a conference paper at ICLR 2026 Table 5: Inference-time parameter counts (in millions). Backbone counts include only the pretrained feature extractor. CBM counts include the CBL and the final classifier. MethodDatasetBackboneBackbone (M)CBLKCBM (M)Total (M) LF-CBMCUBResNet-1811.692080.1511.84 LF-CBMISIC2018ResNet-5025.56350.0725.63 LF-CBMImageNetResNet-5025.56452313.7939.35 DN-CBM RN CUBCLIP RN5038.30819210.0448.34 DN-CBM RN ISIC2018CLIP RN5038.3081928.4546.75 DN-CBM RN ImageNetCLIP RN5038.30819216.5954.89 DN-CBM ViT CUBCLIP ViT-B/1686.2040962.9289.12 DN-CBM ViT ISIC2018CLIP ViT-B/1686.2040962.1388.33 DN-CBM ViT ImageNetCLIP ViT-B/1686.2040966.2092.40 VLG-CBM CA CUBResNet-1811.695350.3812.07 VLG-CBM CA ISIC2018ResNet-5025.56800.1625.72 M-CBM (Ours) CUBResNet-1811.692780.2011.89 M-CBM (Ours) ISIC2018ResNet-5025.56730.1525.71 M-CBM (Ours) ImageNetResNet-5025.5626488.0733.63 EVISUALIZATIONS OFCBLNEURONS In Figures 8, 9, and 10 we show the top–5 activating test images for representative concepts on CUB, ISIC2018, and ImageNet, respectively. These visualizations qualitatively assess whether CBL concepts align with their intended semantics and, when paired with model explanations, help convey whatthe model is actually seeing in the image that influences a prediction. FMORE EXAMPLES OF EXPLANATIONS In this section, we provide additional examples of local explanations of our M-CBMs at NCC=5. The explanations are shown in Figures 11, 12, and 13 respectively from the CUB, ISIC2018, and ImageNet test sets. 18 Published as a conference paper at ICLR 2026 (a) Bird held in human hand(b) White eyebrow stripe (c) Birds on water(d) Black cap with white body (e) Bird in leafy background(f) Very long forked tail (g) Red berries on branches(h) Black head with bright orange breast (i) White patch on black back(j) Black mask around yellow face (k) Reddish-brown throat patch(l) White fan-shaped head patch (m) Iridescent green-blue plumage(n) White wing bars on dark wings (o) Yellow chest with black v-shaped mark(p) Dark, slender waterbird with long neck (q) All-white plumage(r) Black and white longitudinal stripes (s) Birds perched on rocks(t) Large orange-yellow beak Figure 8: Top-5 activating images for CUB concepts 19 Published as a conference paper at ICLR 2026 (a) Blue-white veil(b) Peripheral pigment network (c) Dark brown homogeneous patch(d) Pink-white structureless area (e) Red to purple homogeneous area(f) Clustered blue-gray ovoid nests (g) Small, dark brown dot or globule(h) Scalp lesion with surrounding hair (i) Clustered pink-to-red vascular network(j) Pink scaly plaque with white and brown dots (k) Elongated pink structure with homogeneous tex- ture (l) Parallel lines or streaks within a pigmented lesion (m) Pink-to-light red, structureless, scaly or erythema- tous plaque (n) Pink or light red papule/nodule with a central de- pression or ulceration (o) Atypical irregular dark brown to black blotches with asymmetric borders and variegated pigmentation (p) Pinkish-white structureless areas with multiple small circular or target-like brownish spots (milia-like cysts and comedo-like openings) (q) Clustered white-to-pink round structures (milky or keratotic dots) on a pink background (r) Dark, structureless central area with irregular, dif- fuse pigmentation (s) Irregular brown blotches(t) Asymmetric patch with variegated pigmentation Figure 9: Top-5 activating images for ISIC2018 concepts 20 Published as a conference paper at ICLR 2026 (a) Tiger stripes(b) Hairless dog skin (c) Sunset over water(d) Exposed human skin (e) Colorful umbrella canopy(f) Hexagonal grid structure (g) Hands using laptop keyboard(h) Wooden stick held by a hand (i) Intricate lace or crochet pattern(j) Yellow banana peel with brown spots (k) Animal swimming at the water surface(l) Pairs of footwear placed side by side (m) Erupting lava and glowing volcanic ejecta(n) Person holding an object near their mouth (o) Canon logo text on a circular black surface(p) Printed text and logos on white plastic bags (q) Set dining tables with tablecloths and chairs(r) Dimpled white spherical surface with printed text (s) Creamy pasta with visible bacon or pancetta pieces and black pepper (t) Person bending forward wearing a light-colored shirt Figure 10: Top-5 activating images for ImageNet concepts 21 Published as a conference paper at ICLR 2026 (a) Correctly predictedBarn Swallow (b) Correctly predictedBlue headed Vireowith significant negative reasoning. (c) Correctly predictedCrested Auklet (d) Correctly predictedArctic Tern (e) Correctly predictedRed headed Woodpecker (f) Correctly predictedLoggerhead Shrike Figure 11: Examples of local explanations from CUB test set. 22 Published as a conference paper at ICLR 2026 (a)Melanocytic nevuswrongly predicted asBasal Cell Carcinoma. (b)Melanocytic nevuswrongly predicted asMelanoma. (c) Correctly predictedVascular lesion (d) Correctly predictedActinic keratosis / Bowen’s disease (intraepithelial carcinoma) (e) Correctly predictedMelanocytic nevus (f) Correctly predictedMelanoma Figure 12: Examples of local explanations from ISIC2018 test set. 23 Published as a conference paper at ICLR 2026 (a) Correctly predictedDock (b)Projectilewrongly predicted asVault. (c) Correctly predictedSnorkel (d)Pitcherwrongly predicted asPiggy bank. (e) Correctly predictedWaffle iron (f)Racerwrongly predicted asTow truck. Figure 13: Examples of local explanations from ImageNet test set. 24 Published as a conference paper at ICLR 2026 GADDITIONAL EXPERIMENTS ON DATASET ANNOTATION On the effect of the reference set for annotation quality.Our annotation pipeline provides the MLLM with a reference set of active examples for the target concept before asking it to annotate new samples. The goal is to visually ground the concept, which can be especially important when the textual label alone may not be precise enough (e.g., long tail, rufous crown, or small nevus). To quantitatively assess how much this reference set contributes to annotation quality, we use the human-annotated concept labels available for the CUB dataset and compare the MLLM’s perfor- mance with and without providing the reference images. For each concept, we run two parallel annotations over the same set of randomly selected 25 images: (i) one where the model is first shown 25 positive examples of the concept, and (i) one where the model sees only the concept name and the images to annotate. Ground-truth concept labels from CUB allow us to compute accu- racy in each condition. Of the 312 CUB attributes, we exclude 19 concepts that contain fewer than 50 positive examples, leaving 293 concepts for evaluation for a total of 7325 images to be annotated. Across these concepts, providing the reference grid achieves a measurable improvement as accuracy increases from 65.91% (without reference) to 69.46% (with reference images). This shows that the MLLM effectively leverages the visual examples for better annotation. However, it is important to note that these numbers may not be representative of the absolute annotation quality, given that CUB concept annotations are known to be very noisy. As noted by Koh et al. (2020), each attribute was annotated by a single non-expert crowdworker, and many concepts are fine-grained or subjective (e.g., red belly vs rufous belly). While such inconsistencies limit the maximum attainable accuracy, the relative improvement provides evidence in favor of the reference set being helpful in improving the quality of the annotations. Grid vs single-image for dataset annotation.In the main annotation pipeline, we present the MLLM with a5×5grid of 25 images at once, asking it to output a list of binary predictions corresponding to each image. This batch style strategy is critical for keeping annotation costs man- ageable, especially for large datasets such as ImageNet. However, it is natural to ask whether the MLLM would produce higher-quality annotations if instead queried with one image at a time. To investigate this, we repeat the experiment described in the previous section, but modify the prompt- ing so that each image is shown individually. For every concept, the model receives the same visual reference examples, but is presented with each of the 25 test images in separate calls. We compare the resulting accuracy with the grid-based setup. The single image approach has an accuracy of 74.00%, compared to 69.46% for the grid. This indicates that the MLLM indeed benefits from focusing on one image at a time, likely because it reduces cross-image interference and allows the model to allocate all attention to the visual details of a single example. In principle, this also suggests that prompting granularity is an important design dimension in MLLM-based annotation systems. However, the practical implications of this improvement are highly limited. The single-image method increases the number of model calls by a factor of 25, as each of the 25 images in the original batch now requires its own MLLM request. As an example, for ImageNet this would increase annotation time from days to months and the API cost from USD 370 to USD 9250. As a result, while the single-image approach achieves better annotations, it is currently impractical for most applications. Nevertheless, these findings are encouraging, as they indicate that as MLLMs become more cost-efficient, single-image prompting would be a straightforward way to improve results further. Robustness of MLLM annotation under reference set poisoning.Providing a set of reference images for a target concept generally improves annotation quality, but a potential risk is that if this reference set is noisy (i.e., containing some images that represent different features than the concept), the MLLM might inadvertently internalize these spurious visual features and associate them with the concept. To test whether our annotation pipeline is susceptible to this effect, we design a controlled poisoning experiment in which the reference images for a concept are deliberately contaminated with images from a different concept. We then also contaminate the batch of images to be annotated with images from this secondary concept and measure whether these poisoned images are predicted as positive more often than typical negative examples. This setup provides a direct quantitative assessment of whether the MLLM forms spurious visual associations from the reference examples or remains faithful to the intended concept semantics. The experiment is carried out on the CUB dataset, where ground-truth concept annotations allow measurement of false positive behavior. 25 Published as a conference paper at ICLR 2026 For each target conceptA, we construct two types of negative examples: (i)standard negatives(im- ages that do not containA) and (i)poisoned negatives(images that do not containAbut do contain a different conceptB). For every conceptA, we randomly select a conceptBand contaminate its reference set by replacing 5 out of the 25 reference images with poisoned negatives. Then, during annotation, we also replace 1 standard negative image per annotation batch with a poisoned nega- tive. The remaining pipeline is kept unchanged. We repeat this procedure for all 293 concepts with sufficiently many positive samples and annotate 25 images per concept, resulting in 293 MLLM annotation calls and a total of 7325 images. After annotation, we compute the False Positive Rate (FPR) separately for poisoned negatives and for standard negatives, quantify the difference using Cohen’sheffect size, and test whether the two proportions differ significantly using a two-proportionz-test. Results are reported in Table 6. The results reveal only a small difference between the FPR of poisoned and normal negatives:0.232vs 0.198. The corresponding effect size, Cohen’sh= 0.084, is well below the conventional threshold for a “small” effect (h= 0.2), indicating that the magnitude of this difference is tiny and negligible in practical terms. Consistently, the two-proportionz-test does not find a statistically significant difference between the two FPRs (p-value= 0.157), meaning that the observed effect is statistically indistinguishable from random variation. Taken together, these findings suggest that the MLLM does not meaningfully internalize the spurious visual features introduced through reference-set poisoning. Even when 20% of the reference examples are deliberately contaminated with a conflicting concept, the model’s annotation behavior remains largely stable, indicating that the annotation is primarily guided by the semantic description of the target concept rather than incidental correlations present in the reference examples. Table 6: Results of MLLM-based concept annotation under poisoning of the reference set on the CUB dataset using human-annotated concepts as ground truth. We report the false positive rates (FPR) for poisoned vs normal negative examples, the two-proportionz-test, and the effect size (Cohen’sh). MetricValue FPR (poisoned)0.232 FPR (normal)0.198 Two-proportionz-test (p-value)0.157 Cohen’sheffect size0.084 HACCURACIES UNDERNEC In this section, we report M-CBM accuracies when controlling for NEC instead of NCC. For context, NEC measures the average number of non-zero classifier weights per class (e.g., NEC=5 means that, on average, 5 concepts per class have non-zero weights), while NCC is defined in detail in Section 4 and intuitively measures how many concepts, on average, are needed to explain predictions. NCC can be viewed as a generalization of NEC, as when the coverage level is set toτ= 1, NCC reduces to NEC (see Section H.1 for proof). Because we use a relatively high coverage levelτ= 0.95, the resulting accuracies are not dramatically different (see Table 7). Intuitively, NCC differs from NEC in that it measures sparsity at the decision-level using concept contributions (weights multiplied by activations), rather than just weights. This relaxes the hard cap that NEC implicitly imposes on the effective concept vocabulary. Indeed, with NCC τ = 5, the model may assign non-zero weights to Table 7: Accuracy results at NEC=5 and NEC=avg. DatasetCUBISIC2018ImageNet MetricsAccuracyAccuracyBalanced AccuracyAccuracy SparsityNEC=5NEC=avgNEC=5NEC=avgNEC=5NEC=avgNEC=5NEC=avg M-CBM72.54%73.70%69.38%73.47%68.97%70.45%71.23%73.00% 26 Published as a conference paper at ICLR 2026 many concepts, as long as only about 5 concepts are required, on average, to explain each prediction at coverage levelτ. For instance, M-CBM at NCC 0.95 = 5has 271 concepts with non-zero weights on CUB, 47 on ISIC2018, and 2572 on ImageNet, while at NEC=5 it has 253 on CUB, 29 on ISIC2018, and 2343 on ImageNet. In Figure 14, we provide some examples of concepts that are used by the NCC=5 model but not by the NEC=5 model. (a) Irregular brown blotches(b) Clustered pink-to-red vascular network (c) Scalp lesion with surrounding hair(d) Scattered light brown granular pigmentation (e) Diffuse reddish pigmentation with structureless pattern (f) Pink skin with fine hair and erythematous (reddish) background (g) Birds on water(h) Black plumage (i) Perched on branch(j) Yellow belly or breast (k) Long pointed wings in flight(l) Green leafy background (m) Hands manipulating objects(n) Wooden cylindrical stick (o) Gothic ribbed vaulted ceiling(p) Car dashboard audio control panel (q) Motion blur of fast-moving vehicles(r) Formation of multiple flying objects against the sky Figure 14: Top-5 activating images for CBL neurons that have all zero weights at NEC=5, but not at NCC=5 for all datasets. 27 Published as a conference paper at ICLR 2026 H.1RELATIONSHIP BETWEENNECANDNCC We now formalize the connection between NEC and NCC atτ= 1. Proposition H.1.Let NEC(W F ) = 1 C C X r=1 K X k=1 1 [W F ] k,r ̸= 0 , and letNCC τ be defined as in Section 4. Assume that whenever[W F ] k,r ̸= 0, the corresponding concept logit[g(a (i) )] k is non-zero for all imagesi. This assumption is reasonable, since concept logits are continuous,z-normalized values, so the probability of encountering an exact zero is neg- ligible at machine precision. Then NCC 1 = NEC(W F ). Proof.By definition in Section 4, u (i) k,r = [g(a (i) )] k [W F ] k,r ≥0. If[W F ] k,r = 0, thenu (i) k,r = 0for alli. Under our assumption, if[W F ] k,r ̸= 0, then[g(a (i) )] k ̸= 0 for alli, henceu (i) k,r >0for alli. Thus, for fixedr, u (i) k,r >0⇐⇒[W F ] k,r ̸= 0for alli. For each classr, define m r = K X k=1 1 [W F ] k,r ̸= 0 , the number of concepts with non-zero weight for classr. For any imagei, exactlym r of theu (i) k,r are strictly positive and the remainingK−m r contributions are zero. Letu (i) (1),r ≥ · ≥u (i) (K),r be the sorted contributions. Sorting moves them r positive terms to the front, so K X k=1 u (i) k,r = m r X s=1 u (i) (s),r andu (i) (s),r = 0for alls > m r . By the definition ofκ (i) r (1)in Section 4, it is the smallest number of top-ranked contributions whose sum matches the total contribution P K k=1 u (i) k,r . Since this total is exactly the sum of them r strictly positive terms and all remaining terms are zero, it follows that κ (i) r (1) =m r for alli,r. Substituting into the definition ofNCC 1 , NCC 1 = 1 |D|C X i,r κ (i) r (1) = 1 |D|C X i,r m r = 1 C C X r=1 m r Using the definition ofm r , NCC 1 = 1 C C X r=1 K X k=1 1 [W F ] k,r ̸= 0 = NEC(W F ), which completes the proof. 28 Published as a conference paper at ICLR 2026 IVISUALIZINGSAEFEATURES In Figures 15, 17, and 16 we show the top–5 activating images for SAE neurons on CUB, ISIC2018, and ImageNet, respectively. We also provide their respective description given by the MLLM. These visualizations qualitatively assess whether the MLLM descriptions align with the neuron behavior. We also overlay the activating images with their respective saliency map to show where that neuron was looking. (a) Bird on tree bark(b) White wing patch on dark bird (c) Bird held in human hand(d) All-white plumage gull (e) Bird on ground or rocky surface(f) Small bird perched on a bare branch (g) Orange bill with white facial plumes(h) White eyebrow stripe on a brown bird (i) Outstretched gull-like wings in flight(j) Bright yellow head and chest with black body (k) Green back with pale yellow spotted underparts(l) White patch on the back of a black bird’s neck (m) Bright green body with black face and blue crown(n) Bright yellow throat and upper chest on a songbird (o) Black bird with a bright red and yellow shoulder patch (p) Iridescent blue or green plumage on a medium- sized bird (q) Bold dark streaks on pale underparts(r) Glossy black plumage Figure 15: Top-5 activating images for SAE features in CUB 29 Published as a conference paper at ICLR 2026 (a) Large crowd at an event(b) Deep snow (c) Tiger nose and muzzle(d) Handwritten or printed text on paper (e) Tiger stripes(f) Barbershop window signage (g) Cluster of unripe green bananas(h) Circular speaker grille (i) White rounded plastic surface(j) Stacked audio equipment front panels (k) Analog control knobs and frequency dials(l) Gooseneck with attached cylindrical element (m) Dual vertical antennas on a rectangular electronic device (n) Vertical row of illuminated status indicator lights on electronic device (o) Green printed circuit board with electronic compo- nents (p) Handheld electronic device screen displaying text or interface (q) Rectangular electronic device with visible ports or switches (r) Closed metallic or plastic rectangular device with rounded edges Figure 16: Top-5 activating images for SAE features in ImageNet 30 Published as a conference paper at ICLR 2026 (a) Brown globules(b) Blue-white veil (c) Peripheral pigment network(d) Dark brown homogeneous patch (e) Asymmetric patch with variegated pigmentation(f) Pink-white structureless area (g) Red to purple homogeneous area(h) Clustered blue-gray ovoid nests (i) Brown macule with overlying hair(j) Scalp lesion with surrounding hair (k) Diffuse homogeneous brown pigmentation(l) Clustered pink-to-red vascular network (m) Brown patch at the border between two skin tones(n) Parallel lines or streaks within a pigmented lesion (o) Central dark spot or blotch on a lighter pink back- ground (p) Dark brown circular lesion at the border of a lighter skin region (q) Small, well-circumscribed, dark brown to black central pigmented spot on light skin (r) Clustered white-to-pink round structures (milky or keratotic dots) on a pink background (s) Pink or light red papule/nodule with a central de- pression or ulceration (t) Pink structureless area with scattered brown dots or clods Figure 17: Top-5 activating images for SAE features in ISIC2018 JEXPLANATIONS COMPARED WITH BASELINES 31 Published as a conference paper at ICLR 2026 1. bride in white wedding dress holding bouquet (+4.07) 2. group_of_people_in_celebratory_or_formal_attire (+2.06) 3. men's suit jacket with dress shirt and necktie (+1.62) 4. wedding dress bodice and upper skirt (+1.27) 5. standing person in formal or distinctive clothing (+0.46) 6. Sum of all 2643 other concepts (+0.26) 1. a wedding (+4.34) 2. waiting at the altar (+0.55) 3. groom (+0.35) 4. Sum of all 4520 other concepts (0.0) 1. bridal (+6.42) 2. couple (+2.05) 3. bejing (+1.12) 4. suits (+0.3) 5. parties (+0.17) 6. Sum of all 4091 other concepts (+0.1) M-CBM (Ours) LF-CBM DN-CBM ViT Ground Truth: Bridegroom Predicted: Bridegroom (Conf. 0.753) Predicted: Bridegroom (Conf. 0.135) Predicted: Bridegroom (Conf. 0.948) Figure 18: ImageNet Dataset 1. black mask across eyes (+4.64) 2. gray body with white underparts (+1.75) 3. white wingbars on gray wings (+1.6) 4. dark cap with pale cheek (+0.3) 5. light gray body, black wings (+0.23) 6. Sum of all 273 other concepts (0.0) 1. black cap and white "eyeline" (+1.51) 2. a white breast (+1.48) 3. white and black coloration (+0.9) 4. a white face and underparts (+0.11) 5. Sum of all 204 other concepts (0.0) 1. NOT red shoulders (+1.8) 2. NOT stocky body (+1.35) 3. NOT large black spider (+0.88) 4. NOT small blue green body (+0.77) 5. long black and white tail (+0.6) 6. Sum of all 530 other concepts (+0.85) 1. colors (+0.62) 2. kramer (+0.48) 3. field (+0.46) 4. beige (+0.45) 5. flea (+0.42) 6. Sum of all 4091 other concepts (+0.05) M-CBM (Ours) VLG-CBM CA LF-CBM DN-CBM ViT Ground Truth: Great Grey Shrike Predicted: Great Grey Shrike (Conf. 0.70)Predicted: Great Grey Shrike (Conf. 0.41) Predicted: Great Grey Shrike (Conf. 0.08)Predicted: Scissor tailed Flycatcher (Conf. 0.09) Figure 19: CUB Dataset 32 Published as a conference paper at ICLR 2026 1. atypical irregular dark brown to black blotches with asymmetric borders and variegated pigmentation (+5.0) 2. NOT uniform light-to-medium brown pigmentation with smooth, regular borders (+1.03) 3. blue-white veil (+0.76) 4. multicomponent structure with multiple clustered brown to black dots and globules (+0.75) 5. NOT scattered light brown pigmentation (-0.03) 6. Sum of all 68 other concepts (0.0) 1. NOT dryness or scaling (+1.16) 2. NOT hair follicles (+1.09) 3. NOT Bleeding (-0.38) 4. NOT Scaly or crusted surface (+0.07) 5. NOT Epidermis and dermis layers (+0.01) 6. Sum of all 30 other concepts (0.0) 1. cancer (+1.17) 2. redness erythema (+0.75) 3. NOT sun damaged skin (-0.19) 4. scaly or rough patches (+0.14) 5. NOT scaly or crusted surface (-0.12) 6. pathological entity (+0.08) 7. ulceration in some cases (+0.08) 8. Sum of all 73 other concepts (+0.05) 1. magnesium (+0.05) 2. ted (+0.03) 3. Sum of all 4094 other concepts (0.0) M-CBM (Ours) VLG-CBM CA LF-CBM DN-CBM ViT Ground Truth: Melanoma Predicted: Melanoma (Conf. 0.70)Predicted: Melanoma (Conf. 0.45) Predicted: Vascular lesion (Conf. 0.54)Predicted: Melanoma (Conf. 0.17) Figure 20: ISIC Dataset 33 Published as a conference paper at ICLR 2026 Table 8: Accuracy results at NCC=5 and NCC=avg of M-CBM using an open-source MLLM (InternVL3.5-241B-A28B) for naming and annotation ande5-large-v2for concept merging. DatasetCUBISIC2018ImageNet MetricsAccuracyAccuracyBalanced AccuracyAccuracy SparsityNCC=5NCC=avgNCC=5NCC=avgNCC=5NCC=avgNCC=5NCC=avg M-CBM71.40%74.02%67.33%74.25%69.26%72.20%56.76%68.76% Table 9: Evaluation of concept predictions on the test set for M-CBM using an open-source MLLM (InternVL3.5-241B-A28B) for naming and annotation ande5-large-v2for concept merging. DatasetCUBISIC2018ImageNet Metrics ROC-AUCROC-AUCROC-AUC MacroWorst-10%MacroWorst-10%MacroWorst-10% M-CBM74.49%58.14%74.36%56.53%70.97%59.28% KPERFORMANCE WITH AN OPEN-SOURCEMLLM In this appendix, we evaluate how M-CBM behaves when replacing the proprietary components used for concept naming, dataset annotation, and concept merging with open-source alternatives while keeping the rest of the pipeline unchanged. For this experiment, we useInternVL3.5-241B- A28Bas the MLLM for both naming and annotation (it supports multi-image inputs, so we can preserve our prompting protocol), and we usee5-large-v2to embed and merge concept names with the same similarity-based merging strategy as in the main paper and the same threshold (i.e., 0.98). Table 8 reports the downstream performance obtained with this fully open-source variant. On CUB, we obtain 71.40% accuracy at NCC=5 and 74.02% at NCC=avg; on ISIC2018, we obtain 67.33% and 74.25% accuracy, with balanced accuracy of 69.26% and 72.20%; and on ImageNet, we obtain 56.76% accuracy at NCC=5 and 68.76% at NCC=avg. Since we do not have ground-truth concept annotations, we follow the same proxy evaluation as in the main paper by annotating the test set with the same pipeline and then measuring ROC-AUC of concept predictions against these labels. Table 9 reports the macro-average ROC-AUC across concepts and the average ROC-AUC over the worst 10% concepts. On CUB, the ROC-AUC is 74.49% (macro) and 58.14% (worst 10%); on ISIC2018, it is 74.36% and 56.53%; and on ImageNet, it is 70.97% and 59.28%. Overall, using the open- source MLLM leads to lower concept-prediction quality and lower downstream performance on all three datasets, but the effect is not uniform across them. On CUB and ISIC2018, the degradation is noticeable but relatively limited, whereas on ImageNet it is substantially larger. Qualitatively, we observe that InternVL3.5 tends to produce more generic and highly similar concept names, which increases the amount of merging and reduces the final concept vocabulary. For example, on CUB we obtain 195 final concepts instead of the 278 concepts produced with GPT-4.1, on ISIC2018 we obtain 38 concepts instead of 73, and on ImageNet we obtain 1768 concepts instead of 2648, again substantially fewer than with GPT-4.1. This effect appears to be particularly problematic on ImageNet, where the much larger and finer-grained set of classes seems to require more concepts and higher-quality annotations. This is also consistent with the lower proxy ROC-AUC values for concept prediction. At the same time, the same qualitative trend of lower annotation quality and fewer final concepts is also present on CUB and ISIC2018, suggesting that concept naming and annotation quality are important across all datasets, even when the downstream impact is smaller. One possible explanation for why the downstream drop is smaller on CUB and ISIC2018 is that these datasets may be more prone to information leakage, since the amount of information needed to separate classes may be substantially smaller than in ImageNet. Overall, these results make even more evident that the quality of concept naming and annotation is critical for downstream classification in M-CBM. On a positive note, since GPT-4.1 annotations are themselves far from perfect and the MLLM component is modular, stronger models may further improve concept quality and, in turn, downstream performance. 34