Paper deep dive
ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing
Roshan Prakash Rane, Marco Simnacher, Manuel Pfeuffer, Marc-Andre Schulz, Nys Tjade Siegel, Maximilian Dreyer, Frederik Pahde, Wojciech Samek, Sonja Greven, Kerstin Ritter
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/29/2026, 3:54:46 AM
Summary
The paper introduces ICON (Independent Canonical cONcept) decomposition, a multivariate concept-based explainability method for deep neural networks. Unlike existing univariate methods (like CAVs and linear probes) that suffer from false positives due to spurious correlations and multicollinearity, ICON quantifies the variance explained by each concept while accounting for all other concepts and the outcome. Validated on synthetic data (ToyBrains), skin-lesion models (ISIC 2019), and neuroimaging models, ICON demonstrates superior accuracy in recovering ground-truth concept importance, detecting shortcut learning without false positives, and providing sparse, calibrated explanations.
Entities (10)
Relation Signals (8)
ICON Decomposition â addresses â Shortcut Learning
confidence 95% · ICON recovers concept importance more accurately... isolates the concepts on which a model genuinely relies
ICON Decomposition â validateson â ToyBrains
confidence 95% · On the ToyBrains synthetic image benchmark... ICON recovers the true importance more accurately
ICON Decomposition â validateson â ISIC-2019
confidence 95% · On the ISIC 2019 skin-cancer benchmark... ICON assigns near-zero importance to artifacts the model was never exposed to
ICON Decomposition â uses â Partial Least Squares
confidence 92% · ICON builds on established principles... combining partial least squares (PLS)
ICON Decomposition â outperforms â Linear Probes
confidence 90% · ICON recovers concept importance more accurately than seven alternative baseline methods... linear probe
ICON Decomposition â outperforms â CAVs
confidence 90% · ICON recovers concept importance more accurately than seven alternative baseline methods... CAV-signal
CAVs â suffersfrom â Spurious Correlations
confidence 85% · univariate decoding methods produce false positives by conflating multicollinearities in the data with concept importance
Linear Probes â suffersfrom â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep neural networks often exploit spurious associations in their training data, a failure known as shortcut learning. Concept-based explainability methods screen for shortcuts by testing whether concepts such as a patient's sex or scanner settings can be decoded from a network layer. Because each concept is evaluated in isolation, these methods can mistake correlations between concepts as evidence that the model uses them. We introduce ICON decomposition, which instead quantifies how much of a layer's variance each concept explains after accounting for all other concepts and the outcome. On synthetic data with known ground truth, ICON recovers concept importance more accurately than seven alternative baseline methods. On skin-lesion and brain-imaging models, it isolates the concepts on which a model genuinely relies, quantifies the representation unexplained by any of the supplied concepts, and yields sparse explanations that we validate by retraining and out-of-distribution testing.
Tags
Links
- Source: https://arxiv.org/abs/2608.26083v1
- Canonical: https://arxiv.org/abs/2608.26083v1
Trouble viewing inline? Open PDF directly â
Full Text
142,703 characters extracted from source content.
Expand or collapse full text
ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing Roshan Prakash Rane Affiliation: Hertie Institute for AI in Brain Health, University of TĂŒbingen, TĂŒbingen, Germany Affiliation: Department of Psychiatry and Neurosciences, CharitĂ© - UniversitĂ€tsmedizin Berlin, Berlin, Germany Affiliation: Department of Psychology, Humboldt-UniversitĂ€t zu Berlin, Berlin, Germany Marco Simnacher Affiliation: Chair of Statistics, Humboldt-UniversitĂ€t zu Berlin, Berlin, Germany Manuel Pfeuffer Affiliation: Chair of Statistics, Humboldt-UniversitĂ€t zu Berlin, Berlin, Germany Marc-Andre Schulz Affiliation: Hertie Institute for AI in Brain Health, University of TĂŒbingen, TĂŒbingen, Germany Affiliation: Department of Psychiatry and Neurosciences, CharitĂ© - UniversitĂ€tsmedizin Berlin, Berlin, Germany Affiliation: TĂŒbingen AI Center, University of TĂŒbingen, TĂŒbingen, Germany Affiliation: German Center for Mental Health (DZPG), TĂŒbingen, Germany Nys Tjade Siegel Affiliation: Hertie Institute for AI in Brain Health, University of TĂŒbingen, TĂŒbingen, Germany Affiliation: Department of Psychiatry and Neurosciences, CharitĂ© - UniversitĂ€tsmedizin Berlin, Berlin, Germany Maximilian Dreyer Affiliation: Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute, Berlin, Germany Frederik Pahde Affiliation: Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute, Berlin, Germany Wojciech Samek Affiliation: Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute, Berlin, Germany Affiliation: Department of Electrical Engineering and Computer Science, Technische UniversitĂ€t Berlin, Berlin, Germany Sonja Greven Affiliation: Chair of Statistics, Humboldt-UniversitĂ€t zu Berlin, Berlin, Germany Kerstin Ritter Affiliation: Hertie Institute for AI in Brain Health, University of TĂŒbingen, TĂŒbingen, Germany Affiliation: Department of Psychiatry and Neurosciences, CharitĂ© - UniversitĂ€tsmedizin Berlin, Berlin, Germany Affiliation: TĂŒbingen AI Center, University of TĂŒbingen, TĂŒbingen, Germany Affiliation: German Center for Mental Health (DZPG), TĂŒbingen, Germany Abstract Deep neural networks often exploit spurious associations in their training data, a failure known as shortcut learning. Concept-based explainability methods screen for shortcuts by testing whether concepts such as a patientâs sex or scanner settings can be decoded from a network layer. Because each concept is evaluated in isolation, these methods can mistake correlations between concepts as evidence that the model uses them. We introduce ICON decomposition, which instead quantifies how much of a layerâs variance each concept explains after accounting for all other concepts and the outcome. On synthetic data with known ground truth, ICON recovers concept importance more accurately than seven alternative baseline methods. On skin-lesion and brain-imaging models, it isolates the concepts on which a model genuinely relies, quantifies the representation unexplained by any of the supplied concepts, and yields sparse explanations that we validate by retraining and out-of-distribution testing. keywordsexplainable AI (XAI), concept-based explanations, shortcut learning, spurious correlations, variable importance, partial least squares, model auditing, neuroimaging Deep neural networks (DNNs) are increasingly employed across biomedicine, from high-stakes clinical decision support to fundamental biomedical research Ramsundar et al. (2019). However, a key barrier to their adoption is their tendency to exploit spurious associations and biases in the training data, a behaviour known as âshortcut learningâ Geirhos et al. (2020). For example, skin-cancer classification models have been shown to rely on surgical skin markings or band-aids in the image rather than on the disease-relevant lesions Winkler et al. (2019); Pahde et al. (2023). The same problem pervades biomedical research. For instance, in neuroimaging, a model may predict a brain disease from a participantâs sex or recruitment site rather than from disease-related brain differences Thibeau-Sutre et al. (2022); Rane et al. (2022). Before deploying such models or interpreting their findings, it is crucial to screen them for shortcuts and ensure that they do not rely on spurious factors. Screening for shortcuts requires us to interpret models post-hoc using variables such as data-acquisition settings, disease indicators, or demographics (e.g., sex and age) Poeta et al. (2023). Standard explainable AI (XAI) methods, such as Grad-CAM Selvaraju et al. (2017) and LRP Bach et al. (2015), are insufficient, since they explain a model with low-level features such as individual pixels or image regions. To fill this gap, Concept-based explainability (C-XAI) methods such as linear probes Alain and Bengio (2017) and Concept Activation Vectors (CAVs) Kim et al. (2018); Pahde et al. (2025a); Graziani et al. (2018) have emerged that explain a model in terms of higher-level, human-understandable âconceptsâ Poeta et al. (2023). Linear probes and CAVs are popular because they can be applied post-hoc to any trained model, without needing retraining or access to the training data. Furthermore, they can be used to interpret the model with any high-level concept, provided enough annotations exist in a held-out sample. However, several recent studies across different applications have shown that both of these methods often produce misleading explanations Elazar et al. (2021); Ravichander et al. (2021); Nicolson et al. (2025); Dreyer et al. (2024); Raman et al. (2024); Li et al. (2024); Ramaswamy et al. (2023); Brown and Kvinge (2023) (Supplementary Note 1). In this work, we argue that these failures arise from their framing: they pose C-XAI as a univariate decoding problem. That is, they analyse each concept in isolation from the others, and ask âis this concept predictable from a DNN layerâs representation?â. We identify three major limitations that follow from this framing. (i) Univariate decoding methods produce false positives by conflating multicollinearities in the data with concept importance. Concepts are often correlated with the outcome or with one another, and a univariate method cannot separate a conceptâs importance from the importance of other correlated concepts. In neuroimaging cohorts, for example, demographic, socioeconomic, and clinical variables can exhibit complex inter-dependencies, often associated with both the imaging measurements and the outcome Hyatt et al. (2020); Alfaro-Almagro et al. (2021). Even when the model does not use a concept such as age or household income as a shortcut, a univariate probe or CAV can still assign it a high importance score if the concept merely correlates with something the layer does encode, such as the outcome or another concept the model relies on Ravichander et al. (2021); Adebayo et al. (2022). For example, if the model uses patientsâ sex as a shortcut to predict the disease, then income or BMI can appear important merely by virtue of being correlated with sex. (i) Existing C-XAI scores are not comparable across layers of the same DNN model, nor across continuous and categorical concept types. Multiple studies have independently shown that these methods generate incoherent interpretations across layers, with a concept appearing important at one layer but not at the next Nicolson et al. (2025); Dreyer et al. (2024); Belinkov (2022). Furthermore, these methods offer different metrics for categorical and continuous concepts (e.g., Cohenâs Îș versus R2R^2 for probes Rane et al. (2024b), and TCAV Kim et al. (2018) versus RCAV Graziani et al. (2018) for CAVs). They offer no principled way to reconcile these differences and inconsistencies Nicolson et al. (2025). Finally, (i) the univariate decoding framing cannot say how much of a representation the supplied concepts leave unaccounted for. For example, in neuroimaging, even when our candidate concepts sex, age, income all score highly for a brain-disease model, the model may still be relying on a genuine biomarker beyond these factors. Scoring concepts one at a time gives no way to quantify what the set as a whole accounts for, and so no way to warn a user when the supplied concept set is inadequate. Several fixes have been proposed for these limitations. Signal-CAV, for instance, replaces the decoder in CAVs with a covariance-based estimator Pahde et al. (2025a), a deliberate move away from decoding. Other solutions include forcing concept orthogonality in the model representation Erogullari et al. (2026), or doing follow-up diagnostics such as inspecting multicollinearities and layer-wise consistency Nicolson et al. (2025). Each of these targets one symptom, but none challenges the core univariate framing itself, which generates these issues. We therefore propose a C-XAI method called Independent Canonical cONcept (ICON) decomposition, which instead reframes C-XAI as a multivariate variance-decomposition problem. Rather than asking whether a concept is decodable, ICON asks: âhow much of the variance in a DNN layerâs representation is explained by each concept, given all other concepts and the outcome?â. ICON additionally reports which of the supplied concepts are redundant and share variance with one another. ICONâs reframing addresses all three limitations. ICON tackles (i) by making the multicollinearity among the concepts and the outcome explicit, and then assigning each shared variance to the variable that contributes most strongly to it. This ensures that a concept that is merely correlated with another important concept, or with the outcome, receives no importance, leaving only a few concepts that require further interpretation. ICON resolves (i) by expressing importance as a proportion of the representationâs variance. ICON scores are directly comparable across layers because the explained shares and the unexplained share always sum to one, irrespective of the layerâs dimensionality. Furthermore, explained-variance scores cover both categorical and continuous concepts within one framework. For (i), ICON additionally reports the share of a representationâs variance that none of the supplied concepts explains. A large unexplained share is itself informative, as it indicates that the supplied concepts cannot fully account for what the layer encodes, either because a relevant concept is missing from the set (e.g., a genuine biomarker), or because the encoding is too nonlinear for a linear method to capture. In other words, ICON explanations are calibrated since they provide an explicit statement of how much of the representation is still unaccounted for. ICON builds on established principles of variance partitioning in statistics Grömping (2015); Clouvel et al. (2025); Dinga et al. (2020), combining partial least squares (PLS), which handles high-dimensional model representations Wold (1966), with a Type I (sequential) sum-of-squares allocation Kutner et al. (2005) that reduces false positives from correlated concepts. We validate ICON in three experiments, ranging from a synthetic dataset with known ground truth, to screening for shortcuts in real-world medical models, to auditing open biomedical research models. 1. On the ToyBrains synthetic image benchmark Rane et al. (2024a), where the ground-truth concept importance is known from the data-generating process, ICON recovers the true importance more accurately than classical variance-decomposition methods, probes, or CAVs. ICONâs advantage widens as the available test sample shrinks or the number of correlated concepts supplied grows. 2. On the ISIC 2019 skin-cancer benchmark HernĂĄndez-PĂ©rez et al. (2024), we insert synthetic artifacts at controlled rates and train models to use them as shortcuts. ICON assigns near-zero importance to artifacts the model was never exposed to, and tracks the strength of the shortcut as we increase it. Competing C-XAI methods, in contrast, produce false positives driven by correlations in the test data. 3. Finally, on two open neuroimaging research questions, binge-drinking detection and the brain-age gap, we show that ICONâs explanations are actionable, i.e., specific enough to motivate a follow-up experiment that can confirm or refute them, whereas existing methods produce explanations that are misleading and incomplete. For binge-drinking detection, ICON identifies sex as a shortcut the model relies on, which we confirm by retraining on sex-balanced data. For the brain-age model, ICON surfaces nothing beyond age, whereas a linear probe surfaces numerous age-correlated concepts that we then show, with out-of-distribution tests, to be false positives. 1 Results 1.1 In simulated data, ICON recovers the ground-truth concept importance more accurately than existing methods Figure 1: Simulation dataset setup and results. a, Example 64Ă6464Ă 64 ToyBrains images, rendered from sixteen generative image attributes (kl_k) that control the position, colour and intensity of the shapes. b, The data-generating graph. dl_d carries the true predictive signal into y through ÎČd _l_d; 1c_1 and 2c_2 are confounders acting through ÎČ1 _c_1 and ÎČ2 _c_2. We tune (ÎČd,ÎČ1,ÎČ2)( _l_d, _c_1, _c_2) to generate six different causal scenarios ranging from predominantly confounder-driven to predominantly signal-driven (Table 2), while all the edge weights in grey are kept fixed. c, The trained network. We interpret Lâ2H_L-2, Lâ1H_L-1 and the prediction logit LH_L, over five architectural variants Ă three random seeds Ă six causal scenarios (9090 models in total). d-e, We generate concept importance for these models using n=500n=500 test samples and a randomly drawn set of q=30q=30 concepts. d, Error from the ground truth (TVD, Equation 11), grouped by the causal role of the concept that generates the error. Bars are medians over the 9090 models and error bars denote the 25th to 75th percentiles. The CAV baselines are omitted at LH_L (asterisk) as they cannot be computed at a one-dimensional output. e, Median importance assigned to the concepts with zero ground-truth importance, shown against their spurious correlation r with a parent concept that influences the image. The r=0r=0 point is given by the independent concepts, which have no parent at all. Bands denote the 25th to 75th percentiles over the same 9090 models. No real dataset can reveal how much of a modelâs representation is causally attributable to each concept. Therefore, we use the ToyBrains simulator Rane et al. (2024a), in which one can specify the causal graph that generates the images and the outcome (Figure 1a,b) and derive the ground-truth importance of each concept at each layer by intervening on the causal graph (Methods 3.4). We compare this ground-truth concept importance with the estimates produced by ICON and seven baseline methods, using total variation distance (TVD; 00 is exact recovery, 11 complete disagreement). The baseline methods include three C-XAI methods (linear probes, CAV-classic and CAV-signal) and four classical variable-importance estimators (marginal R2R^2, standardised regression coefficients, partial R2R^2 and Shapley R2R^2; Methods 3.3). Each method receives q=30q=30 concepts drawn from a pool of 100100. Each concept occupies one of five causal roles in the graph (Figure 1d legend and Methods 3.4). The sixth causal role is played by an unobservable image attribute dl_d that carries the true predictive signal and is not part of the concept pool. Beyond dl_d, only 1515 of the 100100 concepts, namely the two confounders and the 1313 concepts encoded in the image, causally influence the model representations. The remaining concepts (shown as pink hatches in Figure 1d) have a ground-truth importance of exactly zero. These are divided into four groups of roughly 2020 concepts each with each group spuriously correlated with one of the 1515 causal concepts at one of four levels,râ0.0,0.2,0.4,0.6râ\0.0,0.2,0.4,0.6\, where r=0r=0 implies independence from every variable in the graph. To ensure our results are robust to different model settings and scenarios, we report medians over 9090 trained models (six causal scenarios Ă five architectural variants Ă three seeds; balanced accuracy 75.8±2.0%75.8± 2.0\%; Extended Data Table 2). In hidden layers, ICON is the most accurate method (Figure 1d). At Lâ2H_L-2 ICONâs error is 0.330.33, against 0.520.52 for the best baseline (marginal R2R^2), 0.690.69 for the linear probe and 0.750.75 for CAV-signal. At Lâ1H_L-1 ICON has 0.230.23 against 0.380.38 for the next best baseline (partial R2R^2). Note that, at the scalar decision logit LH_L (final output of the model), the partial R2R^2 method is more accurate than ICON (0.140.14 versus 0.160.16). This is expected, as at pL=1p_L=1 the multi-response regime that motivates ICONâs PLS step does not arise, and an ordinary multiple regression on a scalar LH_L is well conditioned. ICONâs advantage therefore lies at the high-dimensional internal representations, such as Lâ1H_L-1 and Lâ2H_L-2, that C-XAI methods are most often applied to. ICONâs advantage comes from estimating the confounders accurately and suppressing false positives from collinear concepts (hatched segments, Figure 1d). Decomposing each methodâs error by causal role shows that the baselines lose most ground on the 22 confounders (medium-green hatch in Figure 1d) and the 6060 correlated concepts (medium-pink hatch). For every baseline these two roles together carry 6969â82%82\% of the total error at Lâ1H_L-1. Across the three roles whose true importance is exactly zero (pink hatches; outcome-encoded, correlated and independent concepts) ICON misallocates only 0.060.06 of TVD at Lâ1H_L-1, against 0.160.16 for the best baseline (partial R2R^2) and 0.360.36 for the worst (CAV-signal). This is further confirmed when we observe how each methodâs importance for zero-importance concepts scales with the spurious correlation r (Figure 1e). Between r=0r=0 and r=0.6r=0.6 the spurious importance rises by a factor of 77â12Ă12Ă for the linear probe and 77â17Ă17Ă for marginal R2R^2, but only by 1.21.2â1.3Ă1.3Ă for ICON. Two baselines, partial R2R^2 and the regression coefficients, also show no dose response to r, but they fail in a different way. Already at r=0r=0, that is, concepts independent of every variable in the graph, they assign 33â8Ă8Ă the importance ICON does. No method recovers the true predictive signal exactly, because dl_d is unobservable and every method must approximate it through the outcome y (Methods 3.4). This proxy gap accounts for most of ICONâs own residual error. ICONâs advantage widens as the concept set grows or the sample size shrinks (Extended Data Figure 6). As the number of supplied concepts q grows from 55 to 100100, ICONâs error at Lâ1H_L-1 never exceeds 0.220.22, while every baseline degrades steeply, to 0.720.72 for the linear probe and 0.850.85 for CAV-signal (Extended Data Figure 6a). Each added concept is one more collinear distractor that the univariate baselines cannot reject on its own. Only at the smallest concept sets, where few zero-importance concepts are present, do baselines match ICON (Shapley R2R^2 at qâ€5q†5 and partial R2R^2 at qâ€10q†10). ICON also has the lowest error in the small-sample regime (Extended Data Figure 6b). This matters in practice, since concept annotations are typically available for only a few hundred data points (e.g., in biomedical cohorts). Beyond nâ500nâ 500 ICONâs lead is stable, whereas the linear probe barely improves from n=25n=25 to n=5,000n=5,000 at Lâ1H_L-1. Varying the width of the interpreted layer shifts every methodâs error only modestly and leaves ICON the most accurate at every width (Extended Data Figure 6c). Figure 2: ISIC 2019 experiment setup. a, Example images from the ISIC dataset: the nevus class (NV), NV with a red_color artifact, NV with a band_aid, the melanoma class (MEL), MEL with the inserted microscope lens artifact, and MEL with the inserted microscope plus timestamp artifacts. b,c,d, We build two families of adulterated (synthetic artifact-inserted) dataset variants by inserting the two synthetic artifacts at controlled rates k. b, Inserting the microscope artifact into k%k\% of the MEL images raises the MELâmicroscope correlation from 0.300.30 at k=10%k=10\% to 0.930.93 at k=90%k=90\%. c, Models âłmicro-âkM_micro-k trained on those variants come to rely on microscope to an increasing degree, measured as the gain in balanced accuracy Îâacc between the modelâs own test set containing the artifacts and the clean test set. This shortcut learning accuracy rises from +0.4%+0.4\% at k=10%k=10\% to +10.3%+10.3\% at k=90%k=90\%. d, Adding a timestamp overlay to k%k\% of the images that already carry the microscope artifact raises the microscopeâtimestamp correlation from 0.310.31 to 0.930.93. The timestamp artifact never appears in any training set. The concept and outcome composition of the dataset and the correlations among them are shown in Extended Data Figure 8. Figure 3: ICON detects shortcuts in ISIC skin-cancer classifiers without producing false positives. The y-axis denotes the normalised concept importance ImpÂŻ Imp, the importance assigned to a concept divided by the total importance across all nine artifact concepts and nine outcome classes, at the same layer. Colours denote C-XAI methods and line styles denote the two final layers of the VGG-16 model, Lâ1H_L-1 and LH_L (L=18L=18). a, Specificity to conceptâoutcome collinearity. The clean model âłmicro-00M_micro-00, whose true microscope importance is zero, evaluated on test sets micro-10D_micro-10 through micro-90D_micro-90 of rising microscopeâMEL collinearity. ICON stays below 0.010.01 at both layers, while the three baselines assign 0.050.05â0.120.12. b, Specificity to conceptâconcept collinearity. The shortcut-affected model âłmicro-40M_micro-40 evaluated for timestamp, an artifact absent from every training set, on test sets of rising timestampâmicroscope collinearity. ICON again stays below 0.010.01, whereas the linear probes track the collinearity, CAV-classic swings between 0.0050.005 and 0.110.11 across adjacent test sets, and CAV-signal stays high at 0.090.09. c, Detecting the increasing importance of the shortcut concept. The ten models âłmicro-00M_micro-00 through âłmicro-90M_micro-90, of increasing shortcut reliance, all evaluated on the same micro-10D_micro-10 split, in which the artifact is present at a low rate. Only ICONâs importance tracks reliance, rising from â0.00â 0.00 to 0.440.44; the baselines change by 0.020.02â0.150.15. Dataset construction and model training are described in Methods 3.5. 1.2 In skin cancer classification, ICON detects shortcut learning without false positives The ISIC 2019 skin cancer dataset (https://challenge.isic-archive.com/landing/2019/) HernĂĄndez-PĂ©rez et al. (2024) is a common benchmark for detecting shortcuts with C-XAI methods Poeta et al. (2023); Pahde et al. (2025a); Nicolson et al. (2025); Dreyer et al. (2024). Its images contain naturally occurring artifacts that are known to act as shortcuts, such as surgical skin markings Winkler et al. (2019), band-aids, rulers and camera reflections. Applying ICON to a VGG-16 model trained on these images reaffirms the previously reported findings Pahde et al. (2025b) that these artifacts are indeed learned by the model. ICON shows that camera reflection, band_aid, skin_marker, and red_color are all encoded at the penultimate representation Lâ1H_L-1 (explained variance 9.8%9.8\%, 3.4%3.4\%, 3.0%3.0\%, and 2.8%2.8\% respectively), with reflection, and red_color propagating into the final prediction logit LH_L (4.0%4.0\% and 2.8%2.8\%). To perform a controlled comparison, we additionally insert synthetic microscope lens and timestamp artifacts into the images ourselves (Figure 2a). Inserting a synthetic artifact into a growing share k of melanoma images (MEL class) raises the correlation between the artifact and MEL (Figure 2b). Models trained on these adulterated datasets rely on the artifact as a shortcut to a degree that grows with the adulteration rate k (Figure 2c; Methods 3.5). We write âłmicro-âkM_micro-k for a model trained with the microscope artifact in k%k\% of the MEL images. âłmicro-00M_micro-00 denotes a model trained on the original dataset in which all samples with microscope artifact were removed. Together with the nine outcome classes, every method is supplied with nine artifact concepts of which seven are natural and two are synthetic (Extended Data Figure 8). ICON does not generate false positives caused by collinearities in the data (Figure 3a,b). We test the specificity of different C-XAI methods in two settings (Experiments 2a and 2b in Methods 3.5). In Figure 3a we evaluate a model âłmicro-00M_micro-00 on different test sets with progressively increasing microscopeâMEL correlation. Because this model was never exposed to the artifact during training, it cannot exploit it, so its true importance is zero. ICON assigns <0.01<0.01 importance at the last two layers for all test sets, irrespective of the adulteration rate k, while the baselines assign importance between 0.050.05â0.120.12 (the same picture also holds with the un-normalised scores; Extended Data Figure 7). The baselines fail in one of two ways: the linear probeâs score rises steadily as the spurious correlation increases in the test set, while the two CAV scores stay high and drift non-monotonically. Either way, all three baselines track the test-set statistics rather than the learned behaviour of the model. In Figure 3b we repeat the test for timestamp, an artifact that appears in no training set. We use the model âłmicro-40M_micro-40 and test sets in which timestamp becomes progressively collinear with microscope. ICON again stays <0.01<0.01, while the baselines produce false positives driven by the collinearity. CAV-classic is the least stable, swinging between 0.0050.005 and 0.110.11 between adjacent test sets, consistent with prior reports that CAV importance shifts under changes to the concept examples and the probing dataset Ramaswamy et al. (2023) and under perturbations of the inputs Brown and Kvinge (2023). ICON tracks the degree of shortcut learning more accurately than the baselines (Figure 3c). Across the ten models the accuracy gained from the artifact rises from +0.4%+0.4\% for âłmicro-10M_micro-10 to +10.3%+10.3\% for âłmicro-90M_micro-90 (Figure 2c), so a responsive C-XAI method should report a growing importance for microscope. Only ICON does, with the normalised concept importance rising from â0.00â 0.00 for âłmicro-00M_micro-00 to 0.440.44 for âłmicro-90M_micro-90, while the three baselines show only small increases between 0.020.02â0.150.15. This is further confirmed when we qualitatively compare the normalised layer-wise importance of all methods (Figure 4). ICON shows a distinct difference between the clean âłmicro-00M_micro-00 and the shortcut-driven âłmicro-40M_micro-40 model in the concept importance of microscope (yellow). The microscope artifact is genuinely present in the input images of the test data and is encoded in the early convolutional layers of both models. In the last few prediction layers, however, microscope-related variance remains dominant in the âłmicro-40M_micro-40 model but is filtered out in the âłmicro-00M_micro-00 model. In contrast, the baselines separate the two models only subtly. ICON additionally reports how much of each layer cannot be explained by the supplied concept set (Extended Data Figure 9). This unexplained share is large in the early convolutional layers and shrinks towards the prediction logits, indicating that the early layers encode concepts outside the supplied set or encode them non-linearly or both. No other C-XAI method can offer such a diagnostic. Having shown that ICON detects shortcuts in a controlled setting, we next apply it to two open research questions from neuroimaging where the true concept importances are not known. Figure 4: ICON is the only C-XAI method whose decomposition separates a clean model from a shortcut-affected one. Each row is one C-XAI method, and each stacked bar is that methodâs layer-wise concept importance for the artifact concepts and the nine outcome classes, from an early convolutional layer Lâ17H_L-17 (bottom) to the prediction logits LH_L (top). Within a row, the left panel is the clean model âłmicro-00M_micro-00 and the right panel the adulterated model âłmicro-40M_micro-40, both evaluated on the same test split (n=2,459n=2,459 images), in which the microscope artifact (yellow) is present in 10%10\% of the MEL images. The un-normalised ICON decomposition with the unexplained share retained is shown in Extended Data Figure 9. 1.3 In real-world biomedical research, ICON produces actionable insights Figure 5: On two UK Biobank brain-MRI models, ICON generates sparse, clear insights about what the model does, while the linear probe produces a diffuse, ambiguous explanation. Layer-wise concept importance for the 3030 concepts and the outcome, shown as stacked bars from an input layer at the bottom to the prediction logits at the top. Colours denote the concept categories in the legend. ICON bars are shares of representation variance and sum to one at each layer, with the black segment denoting Ïl2Ï^2_l, the share of that layerâs variance which none of the supplied concepts explains under a linear model. Probe bars are per-concept probe scores (R2R^2 or Cohenâs Îș) normalised for display, and unlike ICON they carry no unexplained term. a, Binge-drinking classification (3D ResNet-18; fold 1 of 5, accuracy 59.4%59.4\%). ICON (top) assigns â0.39â 0.39 of the final-layer variance to participantsâ sex and essentially none to the binge-drinking outcome or other concepts. The probe (bottom) also ranks sex highly but additionally surfaces the outcome itself and several health-related concepts (BMI, EverAddicted, SelfHarm), leaving it unclear whether the model learned only sex or also a binge-drinking phenotype. b, Brain-age prediction (SFCN; fold 1 of 5, MAE 2.892.89 years). ICON (top) assigns â0.63â 0.63 of the final-layer variance to the outcome, age (remaining â0.36â 0.36 unexplained), and surfaces scanner-related concepts only in the early layers. The probe (bottom) assigns importance to age and to many age-correlated concepts, including EmploymentStatus, HouseholdIncome, BMI, sex, and scanner and imaging settings. DNNs are increasingly used in research to find biomarkers of disease from high-dimensional, unstructured biomedical data such as neuroimaging, genomics, or proteomics Eitel et al. (2021); Ramsundar et al. (2019). In these applications, C-XAI methods can be used for model auditing to verify if a model truly learns a biologically-relevant signal or relies on a spurious artifact or sociodemographic bias in the data. In biomedical research, candidate concepts can be numerous and unavoidably correlated, precisely a regime where the current univariate C-XAI methods can fail. In this experiment, we pick two open research questions from neuroimaging and demonstrate how ICON generates sparse explanations that are actionable, whereas existing univariate C-XAI methods such as linear probes do not. Specifically, we investigate the following two open research questions: 1. A recent study reported that an alcohol-misuse phenotype called binge drinking can be predicted from structural brain MRI in a European adolescent cohort Rane et al. (2022); Rane et al. (2023). We test if we can find a similar brain-signature for binge drinking in an older-adult UK cohort Sudlow et al. (2015). Therefore, we ask, âCan a DNN model identify a brain signature of binge drinking in an older adult population?â 2. DNN models are trained to predict age from brain MRI and the prediction error of the model, called the brain-age gap, is widely used as a transdiagnostic biomarker of brain health Cole et al. (2019); Franke and Gaser (2019). It is argued that the brain-age gap captures markers of accelerated brain ageing and hence is a proxy for brain diseases. We ask, âDo brain-age DNN models read any brain-health signal or spurious association beyond chronological age into their predictions, and does that support or challenge the brain-age gap as a brain-health biomarker?â In both tasks, all methods are supplied with 3030 concepts along with the outcome variable in âČC . Including the outcome in the concepts set ensures that a concept is credited only with the representation variance it explains beyond the outcome itself (Methods 3.1). We compare ICON with linear probes but have to omit CAV-based methods because our concept set contains a mixed set of categorical and continuous variables and CAV scores are not directly comparable across them. This is a major limitation of CAV as such mixed data types are expected during model auditing, especially in the biomedical context Graziani et al. (2018). ICON and linear probes can generate different insights for the same DNN model. We therefore test both against independent evidence: a retraining experiment for the binge-drinking model, and resampled out-of-distribution test sets for the brain-age model (Methods 3.6). On the binge-drinking model, ICON identifies that the model uses sex as a shortcut, while the probeâs attribution stays ambiguous (Figure 5a). A 3D ResNet-18 Hara et al. (2018) classifies binge drinkers from controls at 60±0.5%60± 0.5\% accuracy (mean ± s.d. over five cross-validation folds; chance 50%50\%), a modest but above-chance result that, taken at face value, would read as a positive finding. We inspect what concepts the model has learned using ICON and then linear probes. ICON shows that the variance explained by sex increases steadily from the input layer (Lâ20H_L-20) to the prediction logit (LH_L), with sex explaining nearly all of the concept-explained variance in the final layers. The binge-drinking outcome itself explains essentially none. Thus, ICON suggests that the model uses sex as a shortcut for binge drinking (in this cohort sex correlates with the binge-drinking outcome at râ0.23râ 0.23; Extended Data Figure 10). The probe, in contrast, assigns importance not just to sex, but also to the outcome, and several other mental-health concepts (Figure 5a, bottom). This diffuse attribution produces an unclear picture: should we reject this model as entirely shortcut-driven, or has it additionally learned a genuine mental-health signal beyond sex? Retraining on sex-balanced data confirms ICONâs reading: once sex can no longer be exploited, most of the modelâs above-chance accuracy disappears. The retrained model performed close to chance (53.3±0.5%53.3± 0.5\%, mean ± s.d. over five folds, n=1,270n=1,270 hold-out participants), indicating that whatever above-chance accuracy the original model had was carried by the sex confound. This supports ICONâs reading and leaves the probeâs âpossible alcohol-specific signalâ interpretation unsupported. Repeating the task with a different architecture (SFCN Peng et al. (2021), 61.2±0.7%61.2± 0.7\% over five folds) reproduces the same confound (Extended Data Figure 11b), suggesting that the shortcut originates in the data rather than in the modelling choices. On the brain-age model, ICON attributes the prediction almost entirely to age, whereas the probe surfaces many age-correlated concepts (Figure 5b). Our SFCN model Peng et al. (2021) predicts age with a mean absolute error (MAE) of 2.89±0.022.89± 0.02 years and R2=0.750±0.004R^2=0.750± 0.004 (mean ± s.d. over five folds, n=5,000n=5,000 hold-out participants), on par with the published benchmark. ICON attributes most final-layer variance to the outcome, age, with no dominant shortcut of the kind seen in the binge-drinking model (Figure 5b, top). A few other concepts carry tiny, non-zero importance (e.g., the acquisition date of the scan), together explaining under 1% of the variance in y. ICON quantifies that the remaining â36%â 36\% of the output-layer variance is explained by none of the 3030 supplied concepts, flagging either an incomplete concept set or a nonlinear encoding of the supplied concepts. Neither probes nor CAVs provide calibrated explanations of this kind. On the same representations, the probe instead spreads importance across age and many age-correlated concepts such as socioeconomic status (employment status and household income), BMI, and cognitive test performances Hyatt et al. (2020); Alfaro-Almagro et al. (2021). It also assigns importance to sex, scanner and imaging-site settings (Figure 5b, bottom). The concepts the probe flags do not survive independent validation, consistent with them being decodable but unused false positives. From each method, we take the five concepts ranked most important other than age at the final hidden layer, which gives the eight concepts in Table 1. To test whether the model actually uses them, we draw on the participants remaining in the held-out pool of n=11,354n=11,354 participants to build resampled out-of-distribution (OOD) test sets in which a chosen conceptâs correlation with age is removed, and compare the modelâs prediction error against a matched control set that retains the original correlations (Methods 3.6). A concept the model uses as a proxy for age should degrade the error while a merely decodable one should not. Six of the eight concepts admitted a valid OOD set. None of the six showed a robust change in model prediction error (|ÎâMAE|â€0.069| |†0.069 years, none clearing p<0.05p<0.05 in all five folds), which is the pattern expected of a false positive. ICON assigns near-zero importance (<0.001<0.001) to them while the probe does not. Sex and household income, which the probe ranks second and fourth, therefore appear to be encoded in the representation but not used by the model. The two remaining concepts, employment status (the probeâs top concept) and acquisition date (ICONâs top concept), are too collinear with age for a valid OOD set, so for these we measured the age-conditioned partial R2R^2 of the prediction instead. Employment status carries no detectable age-independent signal (partial R2â0R^2â 0), whereas acquisition date, which the probe leaves outside its top five, is the only concept with a small but significant influence (partial R2â2.4%R^2â 2.4\%, p<0.001p<0.001). This is also consistent with ICONâs layer-wise decomposition. A tiny fraction of the scanner-acquisition concepts that dominate the input-layer representation leaks into the output layer (Figure 5b and Table 1). Tracking ICON across training checkpoints shows why (Extended Data Figure 11a). The residual scanner signal is a training residue as the image representations are scanner-dominated at initialisation, and age comes to dominate only as the model converges. Table 1: No concept the linear probe flagged changes the brain-age modelâs error when its correlation with age is removed. For each of the eight top concepts: the change in MAE between the OOD and the matched random-control test set (Î , in years), the ICON and linear-probe importance at the final hidden layer (Lâ1H_L-1) with the conceptâs rank among all 3030 concepts, and the age-conditioned partial R2R^2 of the model output on that concept. Values are mean ± s.d. over the five cross-validation folds. A dash in a rank column means the concept falls outside that methodâs top five and âinfeasibleâ means no OOD set met the matching thresholds (Methods 3.6). Î is assessed per fold with a two-sample z-test approximation (SâE=Ï^â1/nOOD+1/ncontrolSE= Ï 1/n_OOD+1/n_control, Ï^â2.2 Ïâ 2.2 years from the reference hold-outâs |y^âage|| y-age|). A Î is called significant only if it clears α=0.05α=0.05 in all five folds. None does. Partial R2R^2 is assessed with a nested-model F-test and Bonferroni-corrected across the two concepts, and the asterisk marks the one significant value (p<0.001p<0.001). Concept Î ICON Probe Partial R2R^2 (OODâ-control), yr score (rank) score (rank) (age-adj.) Scanner table position â0.069±0.029-0.069± 0.029 0.001 (4) 0.000 (-) - Systolic blood pressure +0.038±0.043+0.038± 0.043 0.001 (5) 0.072 (5) - Alcohol use frequency â0.031±0.016-0.031± 0.016 0.001 (3) 0.007 (-) - Household income +0.024±0.043+0.024± 0.043 0.000 (-) 0.072 (4) - Sex +0.015±0.031+0.015± 0.031 0.000 (-) 0.108 (2) - Snap reaction time +0.010±0.022+0.010± 0.022 0.001 (2) 0.082 (3) - Acquisition date infeasible 0.004 (1) 0.001 (-) 0.024â±0.0020.024^*± 0.002 Employment status infeasible 0.000 (-) 0.360 (1) 0.000±0.0000.000± 0.000 2 Discussion C-XAI methods like linear probes and CAVs have largely re-derived variable importance without drawing on the statistics literature that has studied this problem for decades Grömping (2015). We adapted this literature to the C-XAI regime and developed a novel multivariate C-XAI method, ICON decomposition, which combines PLS Wold et al. (2001) to handle high-dimensional DNN representations and multiple correlated concepts with a Type I sum-of-squares allocation Kutner et al. (2005) to reduce false positives from those correlations. Linear probes and CAVs are popular because they can be used to interpret a model post-hoc, after training, with any user-defined concept, provided enough annotated test samples exist for that concept. ICON preserves these benefits while overcoming several of their documented failures, specifically their high rate of false positives, and scores that are not comparable across model layers or across concept types. ICON achieves this by recasting the question. The existing methods pose C-XAI as a univariate decoding question, âcan this concept be predicted from a given layer?â, whereas ICON poses a multivariate variance-decomposition question, âhow much of a layerâs variance do the supplied concepts and the outcome jointly account for, and how is that share divided among them?â. Moving from univariate to multivariate ensures that the false positives induced by correlations among the concepts, and between the concepts and the outcome are suppressed (Figure 1e, Figure 3a,b). This shift also enables ICON to quantify the share of each layer that none of the supplied concepts can account for. Moving from decoding to variance decomposition makes ICONâs scores comparable across layers and across categorical and continuous concepts, since every score is a share of each layerâs variance and the total share sums to one. These properties of ICON translate into three key practical benefits. First, ICONâs low false-positive rate yields explanations that are sparse, that is, only a few concepts carry appreciable importance (Figure 1d). In the neuroimaging experiments, ICON concentrated importance on a small number of concepts for which we then found supporting evidence with independent tests, whereas the additional concepts the linear probe ranked highest showed no evidence of use (Figure 5, Table 1). Such sparse attributions can be treated as actionable hypotheses for follow-up, testable with out-of-distribution validations, targeted retraining on rebalanced data, or formal confounder analyses as demonstrated in our experiments. Second, layer-wise comparability lets us follow a conceptâs importance across depth, from the input representation to the prediction, enabling us to distinguish a concept that is merely encoded from one that is carried to the final model predictions. The inserted microscope artifact is encoded in the early input layers of both models, but survives into the prediction layers of only the shortcut-affected model. In the binge-drinking neuroimaging experiment the same trajectory is what flags sex as the shortcut the model relies on, which is later confirmed by retraining on sex-balanced data. Finally, the unexplained share Ïl2Ï^2_l guards against overconfidence in the supplied concept set and in the linearity assumption. ICONâs explanations are in this sense calibrated, since they make evident when the supplied set accounts for very little of the representation. We recommend the following practice when using ICON. Supply as many annotated concepts as are available, including the outcome, since a larger set raises the chance of covering the relevant concept and, unlike in the univariate baselines, does not degrade ICONâs accuracy (Extended Data Figure 6a). Note that ICONâs role is to turn an unprioritised set of decodable concepts into a small number of testable candidates, as our experiments illustrate. Treat the concepts ICON nominates as hypotheses for targeted follow-up tests, such as conditional independence tests Simnacher et al. (2026) or partial confounder tests Spisak (2022). Read the trajectory across layers, from the input representation to the prediction layer, rather than any single layer, and check the unexplained share. Finally, ICON is not limited to detecting shortcuts but can also be used to analyse how the concept composition of a modelâs representations changes over training and how it differs between different model architectures (Extended Data Figure 11a,b). Three limitations bound ICONâs claims. First, ICON remains a linear method and rests on the same linear-encoding assumption as the baselines, so a nonlinearly encoded concept is under-attributed and can surface instead in the unexplained share. Second, that unexplained share has two sources, concepts missing from the supplied set and nonlinearly encoded concepts, and we currently do not decompose it into the two. Finally, like the baselines, ICON scores the concepts it is given and does not discover new ones, so the unexplained share reports that something is missing but not what. CAVs are used as directions for steering and editing models post-training, e.g., in large language models Zhang et al. (2025), but they are known to produce unintended collateral effects Erogullari et al. (2026) due to multicollinearities in the data, a problem that ICON explicitly addresses. A future direction could be to test whether steering along ICONâs vectors can reduce such collateral effects. Another natural extension is to relax the linear-encoding assumption by replacing the PLS step with a nonlinear estimator such as kernel PLS. More broadly, the conditions under which ICONâs observational decomposition can be read causally also remain open. Nonetheless, ICON resolves a broader problem in concept-based explainability: existing methods have so far asked the wrong question, making them incapable of distinguishing concept importance that merely reflects multicollinearity in the data distribution from concepts a model actually relies on for its predictions. When ICON is used to audit a model for shortcuts or to generate hypotheses, it returns the few concepts a follow-up experiment can confirm or refute, rather than a long list of decodable concepts. Furthermore, ICON needs only a trained modelâs activations and a few hundred annotated test samples for a stable decomposition (Extended Data Figure 6b), which matches what is typically available in clinical applications and biomedical research cohorts. 3 Methods 3.1 ICON Decomposition Formal definitions: Let us consider a DNN model trained on a dataset ,\X,y\ with n samples, where X denotes the input features (e.g., dermoscopic images or neuroimaging) and y denotes the outcome variable being predicted (e.g., melanoma or a brain disorder). A DNN model with L layers can be expressed as a series of transformations ML:âM11âM2âŻLâ2âMLâ1Lâ1âMLLM_L:X M_1H_1 M_2·sH_L-2 M_L-1H_L-1 M_LH_L, whereby lâânĂplH_l ^nĂ p_l represents the latent representation learned by the model at an intermediary layer l. In C-XAI, the goal is to interpret lH_l using a set of concept variables =[c1,âŠ,cq]C=[c_1,âŠ,c_q] by estimating the importance ImpâĄ(k)Imp(c_k) at layer l. Formal motivation for ICON decomposition: Current C-XAI methods such as linear probes and CAVs estimate the importance of each concept individually (univariate estimation of ImpâĄ(k)Imp(c_k)). It is often the case that ckc_k is correlated with another important concept cjc_j or the outcome variable y. Estimating ImpâĄ(k)Imp(c_k) while omitting the other correlated variables therefore conflates the importance of kc_k with theirs. In a classical regression model, this is the same omitted variable bias that inflates a coefficient when a correlated predictor is left out of the model (Kutner et al., 2005, Ch. 10). Throughout this paper we call a concept a false positive when a C-XAI method assigns it appreciable importance although the modelâs representation does not depend on it. In ICON decomposition, we prevent this bias by controlling for all the supplied concept variables and for the outcome jointly (i.e. multivariate estimation of ImpâĄ(k)Imp(c_k)). We collect them into a single matrix âČ=[,1,2,âŻ,q]âânĂ(q+1),C =[y,c_1,c_2,·s,c_q] ^nĂ(q+1), (1) and define ImpâĄ(k)Imp(c_k) as the variance explained by kc_k in lH_l after controlling for all the other variables in âČC . Preprocessing and notation: Before applying PLS we standardise every variable in âČC and every column of the latent representation lâânĂplH_l ^nĂ p_l to zero mean and unit standard deviation. Step (1) Use PLS to capture latent associations PLS is a multivariate statistical method for finding linear associations between two sets of variables when both sets are high-dimensional and internally collinear, which is exactly the situation we face with the DNN layer representation lH_l and the concept matrix âČC . At each iteration i, the optimization finds weight vectors l,iââplu_l,i ^p_l and l,iââq+1v_l,i ^q+1 that maximize: maxl,i,l,iâĄCovâ(l(i)âl,i,âČ(i)âl,i)=maxl,i,l,iâĄl,iâ€âl(i)â€ââČ(i)âl,i _u_l,i,\,v_l,i\;Cov\! (H_l^(i)u_l,i,\;C (i)v_l,i )\;=\; _u_l,i,\,v_l,i\;u_l,i H_l^(i) C (i)v_l,i (2) where lâ€ââČ is the original cross-covariance matrix at i=1 $ H_l C $ is the original cross-covariance matrix at $i=1$ This optimization yields a latent concept ~l,i=âČ(i)âl,i c_l,i=C (i)v_l,i, called the Independent Canonical Concept (ICON), and a corresponding latent activation vector ~l,i=l(i)âl,i h_l,i=H_l^(i)u_l,i, called the ICON activation vector, that together have the largest covariance attainable. From a CAV practitionerâs perspective, the weight vector l,iu_l,i is the CAV of the latent concept ~l,i c_l,i, and ~l,i h_l,i is that conceptâs score in each sample. The first ICON pair ~l,1,~l,1\ c_l,1, h_l,1\ is computed directly on the original matrices (l(1)=lH_l^(1)=H_l and âČ(1)=âČC (1)=C ). For all subsequent iterations, the optimization is performed on residuals: l(i+1)=l(i)â~l,iâl,iâ€,âČ(i+1)=âČ(i)â~l,iâl,iâ€H_l^(i+1)=H_l^(i)- h_l,i\, Îł_l,i , (i+1)=C (i)- c_l,i\, ÎŽ_l,i (3) where l,iââpl Îł_l,i ^p_l and l,iââq+1 ÎŽ_l,i ^q+1 are the respective loading vectors, that are obtained by performing ordinary least squares projections of each block onto its own score, l,i=l(i)â€â~l,i/(~l,iâ€â~l,i) Îł_l,i=H_l^(i) h_l,i/( h_l,i h_l,i) and l,i=âČ(i)â€â~l,i/(~l,iâ€â~l,i) ÎŽ_l,i=C (i) c_l,i/( c_l,i c_l,i). Since each block is deflated by its own score, all ICON pairs are mutually orthogonal: ~l,iâ€â~l,j=0 h_l,i h_l,j=0 and ~l,iâ€â~l,j=0 c_l,i c_l,j=0 for iâ jiâ j. In the end, this produces a set of m orthogonal ICON activations ~l,1,~l,2,âŠ,~l,m\ h_l,1, h_l,2,âŠ, h_l,m\ that span a low-rank subspace of lH_l, where m is bounded by the rank of the cross-covariance matrix in Equation 2, which in turn satisfies rankâ€minâĄ(pl,q+1)rank†(p_l,\,q+1) Wegelin (2000). In practice, we add pairs one at a time and stop when a new pair adds less than 1%1\% of the total squared cross-covariance, so m is selected adaptively per layer, as is standard practice in PLS Wold et al. (2001). Each ICON activation vector is associated with a corresponding latent concept through an âinner relationâ ÎČi _i Wold et al. (2001) estimated with ordinary least squares (OLS): ~l,i=ÎČiâ~l,i+Ï”i h_l,i= _i c_l,i+ _i. The first ICON pair captures the strongest latent linear association ÎČ1 _1 between lH_l and âČC , and subsequent pairs capture successively weaker, linearly independent associations, since each iteration maximises covariance on the residuals left by the previous one. Step (2) Deriving concept importance by decomposing the variance explained We define the importance of each concept ImpâĄ(k),âkââČImp(c_k),\; _k , as the total variance explained by kc_k in lH_l. To obtain this, we first derive the variance explained by the orthogonal latent concepts ~l,i c_l,i and then decompose this among the original concept and outcome variables using Type I Sum of Squares. (2a) Deriving variance explained by the latent concepts ~l,i c_l,i: For each ICON pair, we can derive the variance explained by the latent concept ~l,i c_l,i in lH_l as follows, l=âi=1m~l,iâl,iâ€+l _l= _i=1^m h_l,i\, Îł_l,i +E_l where l,i Îł_l,i are the loadings at iteration i and lE_l is the residual after repeated subtraction in Equation 3. Applying the variance operator on both sides, Varâ(l)=âi=1mâl,iâ2âVarâ(~l,i)+Varâ(l) (H_l)= _i=1^m\| Îł_l,i\|^2\ \,Var( h_l,i)+Var(E_l) The cross-variance terms vanish since â~l,iâ€â~l,j=0â for âiâ jâ, by design. cross-variance terms vanish since h_l,i h_l,j=0 for iâ j, by design. Substituting the inner relation, Varâ(l)=âi=1mâl,iâ2âVarâ(ÎČiâ~l,i+Ï”i)+Varâ(l) (H_l)= _i=1^m\| Îł_l,i\|^2\ \,Var( _i c_l,i+ _i)+Var(E_l) Since ÎČi _i is estimated by OLS, we assume Covâ(~l,i,Ï”i)=0Cov( c_l,i,\, _i)=0, and therefore Varâ(l)=âi=1mÎČi2ââl,iâ2âVarâ(~l,i)+âi=1mâl,iâ2âVarâ(Ï”i)+Varâ(l) (H_l)= _i=1^m _i^2\| Îł_l,i\|^2\ \,Var( c_l,i)+ _i=1^m\| Îł_l,i\|^2Var( _i)+Var(E_l) Dividing by Var(l)(H_l) on both sides, 1=âi=1mÎČi2ââl,iâ2âVarâ(~l,i)Varâ(l)âImpâĄ(~l,i)+âi=1mâl,iâ2âVarâ(Ï”i)+Varâ(l)Varâ(l)âÏl2â(residual variance) 1= _i=1^m _i^2\| Îł_l,i\|^2\ \,Var( c_l,i)Var(H_l)_Imp( c_l,i)+ _i=1^m\| Îł_l,i\|^2Var( _i)+Var(E_l)Var(H_l)_Ï^2_l(residual variance) Thus, we obtain a clean decomposition by defining the importance of each ICON as the ratio: ImpâĄ(~l,i)=ÎČi2ââl,iâ2âVarâ(~l,i)Varâ(l)â, such that ââi=1mImpâĄ(~l,i)+Ïl2=1 ( c_l,i)= _i^2\| Îł_l,i\|^2Var( c_l,i)Var(H_l)\;\;, such that _i=1^mImp( c_l,i)+Ï^2_l=1 (4) The residual variance Ïl2Ï^2_l is the share of the representationâs variance that cannot be assigned to any concept or to the outcome variable in âČC under the linearity assumption. (2b) Decompose importance among original concepts Tackling the non-identifiability challenge. Although the ICONs are linearly independent of one another by construction, within each ICON the original variables that form it, ~l,i=vl,i,1â1+vl,i,2â2+âŻ+vl,i,qâq+vl,i,q+1â c_l,i=v_l,i,1\,c_1+v_l,i,2\,c_2+·s+v_l,i,q\,c_q+v_l,i,q+1\,y, can be multicollinear. Multicollinear variables share overlapping variance that cannot be uniquely attributed to any one of them, a fundamental constraint known as non-identifiability (Kutner et al., 2005, Ch. 10). In such cases, a reasonable solution is to choose a shared-variance allocation strategy that best serves the goals of our method Grömping (2015). Since our goal is to screen for shortcut variables, we must allocate the total variance explained by the ICON ImpâĄ(~l,i)Imp( c_l,i) across the variables in âČC such that the dominant variables, that is, those with the largest squared weights vl,i,k2v^2_l,i,k within that ICON, absorb any shared variance from variables that are weaker contributors, reducing false positives during the screening. Thus, we decompose the ImpâĄ(~l,i)Imp( c_l,i) using the sequential (Type I) Sum of Squares algorithm, where the order is defined by the squared weights vl,i,k2,âkâ[1,âŻ,q+1]v^2_l,i,k\,,\;â kâ[1,·s,q+1]. Type I sum of squares. Let Ïi _i denote the ordering of the variables that compose an ICON ~l,i c_l,i, such that vl,i,Ïiâ(1)2â„vl,i,Ïiâ(2)2â„âŻâ„vl,i,Ïiâ(q+1)2v^2_l,i, _i(1)â„ v^2_l,i, _i(2)â„·sâ„ v^2_l,i, _i(q+1). Following this ordering, we sequentially regress the ICON activation vector ~l,i h_l,i on the variables in âČC . This yields a set of incremental sums of squares ÎâSSi,Ïiâ(1),âŠ,ÎâSSi,Ïiâ(q+1)\ _i, _i(1),âŠ, _i, _i(q+1)\, where ÎâSSi,Ïiâ(j) _i, _i(j) is the additional variance in ~l,i h_l,i explained by cÏiâ(j)c_ _i(j) after accounting for all higher-ranked variables cÏiâ(1),âŠ,cÏiâ(jâ1)c_ _i(1),âŠ,c_ _i(j-1). These incremental sums of squares add up to the variance in ~l,i h_l,i that âČC explains, so dividing each by their total gives proportions, which we then scale by ImpâĄ(~l,i)Imp( c_l,i). We allocate by ÎâS rather than by vl,i,k2v^2_l,i,k directly because the weights describe how the ICON was built out of âČC , whereas the quantity we want to allocate is variance in ~l,i h_l,i. The importance of concept ckc_k within ICON ~l,i c_l,i is: ImpâĄ(ck,~l,i)=ÎâSSi,kâj=1q+1ÎâSSi,Ïiâ(j)â ImpâĄ(~l,i)Imp(c_k,\, c_l,i)= _i,k _j=1^q+1 _i, _i(j)·Imp( c_l,i) (5) The total importance of a concept ckc_k across all ICONs is then: ImpâĄ(ck)=âi=1mImpâĄ(ck,~l,i)Imp(c_k)= _i=1^mImp(c_k,\, c_l,i) (6) This formulation gives the ICON importance scores four properties that directly tackles the three limitations of existing C-XAI metrics we discuss in the Introduction: 1. Bounded and sum-constrained: ImpâĄ(ck)â[0,1]Imp(c_k)â[0,1] and âkImpâĄ(ck)+Ïl2=1 _kImp(c_k)+Ï^2_l=1, enabling direct comparison across concept types and layers. 2. Transparent about unexplained variance: Ïl2Ï^2_l quantifies how much of the latent representation lies outside the provided concept set and linear assumptions, guarding against overconfidence in an incomplete concept set Poeta et al. (2023), or over-reliance on the linearity assumption CrabbĂ© and van der Schaar (2022). 3. Transparent about entangled concepts: The squared weights vl,i,k2v^2_l,i,k from step 1 record which concepts share variance within each ICON, making multicollinear groups explicit to the practitioner regardless of how the sequential allocation distributes that shared variance. 4. Down-weights weaker, correlated concepts: The dominant variable within each ICON absorbs the variance it shares with weaker correlated ones. This strategy reduces false positives and reveals the variables that drive shortcut learning more accurately. 3.2 Visualising ICON decompositions as layer-wise importance bars ICON and linear-probe importances are visualised as stacked horizontal bars, one stack per interpreted layer, ordered from an input layer at the bottom to the prediction logits at the top and coloured by concept category (for example, Figure 3 and Figure 5). For ICON the bars form a variance partition: their lengths sum to one at each layer (Equation 4), and the fraction of representation variance not explained by any supplied concept, Ïl2Ï^2_l, is drawn as a separate unexplained segment. For the linear probe the bars are per-concept probe scores (cross-validated R2R^2 or Cohenâs Îș) normalised for display (Equation 8); unlike ICON they do not form a variance partition and carry no unexplained term, and only variables exceeding 5% normalised importance are named, although all contribute to the bars. 3.3 Experiment setup Comparing with baseline methods. Across three experiments, we compare ICON against two groups of baseline methods. The first group consists of established C-XAI methods. 1. Linear probes (ImpprobeImp_probe) Alain and Bengio (2017): For each kââČc_k , we fit a logistic (categorical) or linear (continuous) regression predicting kc_k from lH_l. We perform three-fold cross-validation and use the mean test fit quality as the importance score (R2â[0,1]R^2â[0,1] for continuous, Cohenâs Îșâ[0,1]Îșâ[0,1] for categorical, where 00 signifies chance for both). Linear probes are the most widely used C-XAI method, e.g., to assess representation quality in self-supervised models Chen et al. (2020) or detect linguistic structure in language models Hewitt and Liang (2019). 2. CAV-classic (ImpCAV-classicImp_CAV-classic) Kim et al. (2018): A concept activation vector (CAV) is a direction in the layerâs activation space, obtained as the direction of a learned linear probe. Rather than using probe accuracy as the importance score, CAV-classic uses a sensitivity test on the CAV direction which measures how much the modelâs output changes when the layerâs activations are perturbed along it. Following Kim et al. Kim et al. (2018), the score is the fraction of test samples whose directional derivative along the CAV is positive, which equals 0.50.5 under no association; we report its symmetric magnitude |qCAVâ0.5|/0.5|q_CAV-0.5|/0.5, as in Pahde et al. Pahde et al. (2025a), so that the score is a non-negative effect size on the same footing as the other importance measures. CAV-based methods have been applied to detect shortcuts and biases in biomedical applications Pahde et al. (2025b) and to interpret language models Zhang et al. (2025). 3. CAV-signal (ImpCAV-signalImp_CAV-signal) Pahde et al. (2025a): Rather than fitting a probe to obtain the CAV direction, CAV-signal defines the CAV as the difference between the mean activation in lH_l when k=1c_k=1 and when k=0c_k=0 (defined for binary kc_k). Pahde et al. Pahde et al. (2025a) argue that this simpler signal direction makes CAVs less susceptible to confounding by correlated concepts encoded in lH_l. As with CAV-classic, the importance score is the symmetric magnitude |qCAVâ0.5|/0.5|q_CAV-0.5|/0.5 of the directional-derivative statistic, computed along this signal direction. The second group consists of classical statistical methods for variable importance in regression (see Grömping Grömping (2015) for a review), which serve as alternatives to ICONâs PLS-based variance decomposition. 5. Marginal R2R^2 importance (Impmarg-âR2Imp_marg-R^2): The R2R^2 of a univariate regression of lH_l on kc_k alone, ignoring all other concepts. This captures the marginal association between kc_k and lH_l, but when concepts are correlated their shared variance is attributed to each concept individually, inflating importance scores. 6. Standardised regression coefficients (ImpcoefImp_coef): We fit a multi-output linear regression, l=âČâ+,ââ(q+1)Ăpl,âânĂpl,H_l=C W+E, ^(q+1)Ă p_l,\;E ^nĂ p_l, (7) which decomposes into plp_l independent linear regressions, one per output dimension. We use the mean squared coefficient 1plââj=1plWk,j2 1p_l _j=1^p_lW_k,j^2 as the importance of concept kc_k. This accounts for the presence of other concepts, but estimated coefficients are known to be numerically unstable under multicollinearity Grömping (2015). 7. Partial R2R^2 importance (Imppart-âR2Imp_part-R^2): The drop in R2R^2 when kc_k is removed from the full model (Equation 7), i.e., R2â()âR2â(âk)R^2(W)-R^2(W_ k), where âkW_ k is the model refitted without kc_k, and R2R^2 is the coefficient of determination averaged across the plp_l output dimensions. This corresponds to Type I (âlastâ) sum of squares Grömping (2015), quantifying the unique variance explained by kc_k after accounting for all other concepts. Under strong multicollinearity, this unique contribution may shrink to near zero even for genuinely important concepts. 8. Shapley R2R^2 importance (ImpshapImp_shap): The Shapley value Shapley (1953) of kc_k is the average marginal contribution of kc_k across all orderings in which it could enter a Type I sum-of-squares decomposition, and is widely regarded as the most principled regression-based importance measure because shared variance among correlated concepts is distributed symmetrically Shapley (1953); Grömping (2015). We estimate it from the multivariate R2R^2 in Equation 7 using the LMG decomposition Grömping (2015). Normalised importance. Importance scores from different methods, such as ICON, CAVs and linear probes, have different natural scales and the literature does not provide a unified notion that makes these importance scores directly comparable Poeta et al. (2023). We place all the methods on a common simplex by normalising their importance scores over the supplied variable set âČC : ImpÂŻm(l)â(k)=Impm(l)â(k)âj=1q+1Impm(l)â(j). Imp_m^(l)(c_k)= Imp_m^(l)(c_k) _j=1^q+1Imp_m^(l)(c_j). (8) This places all methods on the same fixed budget, and reduces the effect of layer dimensionality on the raw scores. We also report the un-normalised scores alongside the normalised ones wherever a conclusion could depend on this choice (e.g., Extended Data Figure 7). 3.4 Experiment 1: Simulation experiments with ToyBrains ToyBrains Dataset configuration. ToyBrains Rane et al. (2024a) generates 64Ă6464Ă 64 RGB images (X) whose visual attributes are specified by sixteen generative variables =1,âŠ,16L=\l_1,âŠ,l_16\ (Figure 1a). Every path into the image runs through one of the sixteen generative image attributes L. We also generate a binary outcome variable y and a pool of 100100 concepts =1,âŠ,100C=\c_1,âŠ,c_100\. The image attribute variables L are considered unobservable to C-XAI estimations. The data generation process is specified with a directed acyclic graph where the variables ,,\L,C,y\ form the nodes and their causal coefficients form the edges. Every variable is discrete and, in the absence of any incoming edge, uniformly distributed over its states, while an edge coefficient acts by shifting the probability of the target variable through a logistic link. For example, if the target variable z is binary then ÎČ=0ÎČ=0 has no effect as it keeps PâĄ(=1)=50%P(z=1)=50\% as before, while ÎČ=2.20ÎČ=2.20 sets PâĄ(=1)=90%P(z=1)=90\%. Variables with a fixed number of ordinal states are used to emulate continuous variables. For this experiment, we design a graph in which the image and the outcome are related through one âtrueâ signal pathway and two confounder-driven pathways (Figure 1b). A specific attribute, dl_d, carries the true signal, and the remaining fifteen are assigned at random (without replacement) to fifteen concepts that influence the image, so no two concepts share an attribute. Every variable in the graph plays one of the following six causal roles: âą True predictive signal (one variable): This signal is carried by the image attribute dâl_d , which drives a path into both the image and the outcome, âdâX _d . Since the image attributes are unobservable to C-XAI methods, every method must use the outcome y as its observable proxy to estimate the effect size of this pathway. âą Confounder (22 concepts): 1c_1 and 2c_2 each drive a path into both the image and the outcome, âiâaây _i _a for iâ1,2iâ\1,2\, where al_a is that conceptâs assigned attribute. âą Encoded in image (1313 concepts; e.g., jc_j, kc_k, lc_l in Figure 1b): These concepts have a path into the image and none into the outcome, iâaâc_i _a . Thus, they are genuinely visible in the image but irrelevant to the prediction. âą Encoded in outcome (33 concepts): These concepts have only a path into y and none into X, iâc_i for iâ3,4,5iâ\3,4,5\. âą Correlated concept (6060 concepts; sc_s, tc_t in Figure 1b): These concepts have no edge of their own into X or y. They are instead resampled to correlate with one of the 1515 image-influencing concepts above iâic_i, that is, the 1313 image-encoded concepts and the two confounders, as iâiâiâaâc_i _i _a . Each of these concepts is assigned one of three correlation strengths, râ0.2,0.4,0.6râ\0.2,0.4,0.6\, with twenty concepts at each strength. This enables us to study the effect of r on each method (Figure 1e). âą Independent concept (2222 concepts; uc_u, vc_v): The remaining concepts have no edge into any variable and are independent of every other variable in the graph. These serve as the r=0r=0 reference in Figure 1e. Following from this causal graph, the sampling of the binary outcome is given by, PâĄ(=1)=ÏâĄ(ÎČdâd+ÎČ1â1+ÎČ2â2+âi=35ÎČiâi)P(y=1)\;=\;Ï\! ( _l_dl_d+ _c_1c_1+ _c_2c_2+ _i=3^5 _c_ic_i ) (9) where Ï is the logistic function. We alter the data-generating process and generate six different causal scenarios by varying ÎČd _l_d, ÎČ1 _c_1, and ÎČ2 _c_2, while keeping all other edges in the graph fixed. These settings range from a purely confounder-driven regime (ÎČd=0 _l_d=0) to predominantly signal-driven regimes in which one of the two confounders is switched off, as listed in Extended Data Table 2. For each of the six settings we generate 5,0005,000 training and 5,0005,000 test samples. DNN models are trained on the training split, while the test split is used to generate C-XAI explanations and compute the ground truth. By default, each methodâs explanations are computed from an independent random subsample of n=500n=500 test images, whereas the ground truth is always computed on the full test split. Each method is supplied with the same q=30q=30 concepts for each model checkpoint, randomly drawn from a pool of 100 concepts stratified by causal role, ensuring that all causal roles are represented. DNN Models. We train a DNN with three convolutional blocks (32â64â12832â 64â 128 channels, each with 3Ă33Ă 3 convolution with ReLU, batch normalisation, 2Ă22Ă 2 max pooling) followed by three fully connected layers referred to as Lâ2H_L-2, Lâ1H_L-1 and LH_L (Figure 1c), where LH_L is the final scalar decision logit (pL=1p_L=1). We perform C-XAI interpretation at these three last layers of the model. Training uses SGD with a learning rate of 0.010.01, batch size 128128, and up to 100100 epochs with early stopping on the validation loss (patience 1515). To ensure that our experimental conclusions are not specific to one architecture or one random initialisation, we train five architectural variants that differ in the widths of the two interpreted layers (Lâ2,Lâ1)(H_L-2,H_L-1) as (512,128)(512,128), (1024,256)(1024,256), (2048,512)(2048,512), (4096,1024)(4096,1024) and (8192,2048)(8192,2048), and repeat each variant with three random seeds. This gives 1515 models per causal scenario and 9090 models across the six scenarios. Every importance and error value we report in this experiment is a median over all 9090 models, with bands showing the 2525th to 7575th percentile dispersion across models. Balanced accuracy on the test split is 75.8±2.0%75.8± 2.0\% (range 70.770.7â79.1%79.1\%), and varies negligibly across the five architectural variants (75.575.5â76.0%76.0\%). Ground truth. C-XAI methods score each concept separately at every layer, so our ground truth must be layer-wise too. We obtain it by holding the trained weights fixed and intervening on the causal graph in the test distribution (full test split, n=5,000n=5,000). We define the ground-truth importance of a generative variable z at layer l as the share of representation variance accounted for by intervening on z: Imptrue(l)â()=VarâĄ(âĄ[~lâŁdoâĄ()])VarâĄ(~l)Imp^(l)_true(z)\;=\; Var\! (E [ H_l (z) ] )Var ( H_l ) (10) where ~l H_l denotes the activation at layer l, standardised dimension-wise, and VarâĄ(~l)=âjVarâĄ(~l,j)Var( H_l)= _jVar( h_l,j) denotes the total variance of the standardised representation. Here, doâĄ()do(z) denotes a causal intervention operator Pearl et al. (2016), so the numerator is the variance in lH_l accounted for by varying z across its states while keeping the remaining variables in the image-generating graph fixed. Only the variables that have a causal path into X can influence the variance of the modelâs representations lH_l, since doâĄ()do(z) severs the edge from its parent. 8585 of the 100100 concepts do not have a causal path into X. This includes the 6060 correlated concepts, 2222 independent concepts, and 33 encoded-in-outcome concepts. Therefore, we simply set their ground-truth importance to zero analytically. For the remaining sixteen variables that do have a causal path into X, the back-door criterion applies Pearl et al. (2016). These include, the true signal dl_d, the two confounders and the 1313 image-encoded concepts. The back-door criterion applies to them because they are each sampled independently. None of them have a causal parent, so no back-door path runs from any of them into lH_l. Therefore, âĄ[~lâŁdoâĄ()]=âĄ[~lâŁ]E[ H_l (z)]=E[ H_l ], and we can estimate Imptrue(l)â()Imp^(l)_true(z) directly from the observational test split. Error from the ground truth. We place all the methods and the ground truth on a common simplex by normalising their scores over the variable set âČC (denoted as ImpÂŻ Imp; Equation 8). This makes comparing a distance between them well defined. Then we compute the error using the TVDTVD metric: TVDm(l)=12ââkââČ|ImpÂŻtrue(l)â(k)âImpÂŻm(l)â(k)|.TVD^(l)_m\;=\; 12 _c_k |\, Imp^\,(l)_true(c_k)- Imp^\,(l)_m(c_k)\, |. (11) Since both scores sum to one, TVDTVD is bounded in [0,1][0,1] where 00 denotes exact recovery and 11 denotes complete disagreement, irrespective of the value of q. Thus our error metric is directly comparable across methods, causal scenarios and layers of differing dimensionality. One caveat applies to the comparison with CAV and probe baselines which provide a concept ranking and not a variance decomposition like all of the other baseline methods and the ground truth. Thus, their absolute TVD must be interpreted as only an upper bound on their agreement with the ground truth. Ablation experiment sweeps. We perform three ablation experiments in which we sweep three settings fixed above, the sample size n, the concept-set size q and the width of the interpreted layer, around its default. First, we sweep qâ5,10,15,20,30,40,50,60,70,80,90,100qâ\5,10,15,20,30,40,50,60,70,80,90,100\ at fixed n=500n=500 (Extended Data Figure 6a). Next, we sweep nâ25,50,100,250,500,1,000,2,500,5,000nâ\25,50,100,250,500,1,000,2,500,5,000\ at fixed q=30q=30, with each n drawn independently (Extended Data Figure 6b). Finally, we sweep across five variants of layer dimensionality for the layers (Lâ2,Lâ1)(H_L-2,H_L-1) at fixed n=500n=500 and q=30q=30 (Extended Data Figure 6c), to check that our conclusions are not specific to one layer dimensionality. 3.5 Experiment 2: Skin cancer detection The ISIC 2019 Challenge dataset consists of n=24,536n=24,536 dermoscopic images of skin conditions (see examples in Figure 2a), labelled into nine different classes such as the benign melanocytic nevus class (NV, n=12,750n=12,750) and the malignant melanoma class (MEL, n=4,006n=4,006). Additionally, we have annotations for q=9q=9 spurious artifact concepts in these images, including 7 naturally occurring artifacts such as the presence of band_aid (n=184n=184), ruler (n=260n=260), and other camera artifacts, plus 2 synthetic artifacts (microscope, timestamp; shown in green in Extended Data Figure 8a) that we insert into the dataset for controlled experiments. We exclude all n=795n=795 samples containing the natural microscope lens artifact from the clean dataset micro-00D_micro-00, so that âłmicro-00M_micro-00 has no exposure to microscope artifacts during training. All adulterated dataset variants are constructed by adding synthetic microscope artifacts to this microscope-free clean dataset. Extended Data Figure 8a shows the distribution of the nine outcome classes and the nine artifact concepts, and Extended Data Figure 8b shows the Pearson correlations between these variables. The concept red_color is prevalent in the benign NV class (r=0.22r=0.22), ruler in the MEL class (r=0.18r=0.18), and reflection in the benign BKL class (r=0.16r=0.16). DNN Model training and C-XAI explanations: We split the dataset into a fixed 80% training (n=19,622n=19,622), 10% validation (n=2,455n=2,455), and 10% test (n=2,459n=2,459) sets, stratified by the outcome. We use the VGG-16 architecture Simonyan and Zisserman (2015) pretrained on ImageNet as the backbone of the DNN model. We fine-tune the model on the training split using SGD with a learning rate of 0.0010.001, batch size of 6464, and cross-entropy loss. The learning rate is reduced by a factor of 0.10.1 at epochs 100 and 150 over a total of 200 training epochs. We use the validation split to monitor convergence and select the final checkpoint. The clean model âłmicro-00M_micro-00 achieves a balanced accuracy of 85.0% on the validation set, comparable to published benchmarks for VGG-16 on ISIC HernĂĄndez-PĂ©rez et al. (2024). C-XAI explanations are generated on a separate test split and mainly displayed for three layers: the prediction logits (LH_L), and two preceding hidden layers from the classifier head (Lâ1H_L-1, Lâ2H_L-2). Using synthetic artifacts to perform controlled experiments: In the ISIC dataset, we can also synthetically insert realistic-looking artifacts such as microscope lens or timestamp (refer to the rightmost examples in Figure 2a). This allows us to perform controlled experiments similar to Pahde et al. Pahde et al. (2025b), whereby we can induce a spurious correlation between a concept (e.g. microscope) and an outcome class (e.g. MEL) by systematically inserting the artifact in some of the samples belonging to the outcome class. We denote the original ISIC dataset, excluding all samples with real microscope artifacts (n=795n=795) as micro-00D_micro-00 and the adulterated dataset as, e.g. micro-âkD_micro-k, where k is the percentage of the MEL samples in which the microscope artifact was inserted. DNN models trained on adulterated datasets are denoted as, e.g. âłmicro-âkM_micro-k. These models are likely to use the synthetic concept as a shortcut for predicting the correlated outcome class, depending on the value of k. The extent of the shortcut learned by âłmicro-âkM_micro-k can be quantified as Îâacc=accâ(micro-âk)âaccâ(micro-00) =acc(D_micro-k)-acc(D_micro-00), i.e., the difference in the balanced accuracy when the model is evaluated on its own (adulterated) test set versus on the test set of the clean (unadulterated) data. Therefore, with Îâacc we measure how much accuracy the model draws from the artifact by removing the artifact from the test images. We create nine dataset variants micro-10D_micro-10 through micro-90D_micro-90 by adding the microscope artifacts at increasing rates to the MEL samples, as shown in Figure 2b. We then train DNN models on all 10 dataset variants, including the original micro-00D_micro-00 to obtain 10 models with different levels of shortcut learning. The Îâacc increases monotonically from +0.4%+0.4\% (âłmicro-10M_micro-10) to +10.3%+10.3\% (âłmicro-90M_micro-90), approaching the theoretical maximum of 1/9â11.1%1/9â 11.1\% (since balanced accuracy averages per-class-recall across 9 outcome classes), as shown in Figure 2c. We use this controlled experiment setup to compare ICON against three established C-XAI methods: linear probes, the CAV-classic Kim et al. (2018) and CAV-signal Pahde et al. (2025a). We design three experiments that evaluate specificity and responsiveness of C-XAI methods at detecting shortcuts. In Experiments 2a and 2b, we test whether C-XAI methods produce false positives when conceptâoutcome or conceptâconcept collinearity changes in the test data. Such covariate shifts are common in practice: spurious correlations present in the training data may not be present in test data at a clinical site where the model is deployed Geirhos et al. (2020). In Experiment 2c, we test if these methods are able to capture increasing degrees of shortcut learning. Experiment 2a: false positives from concept-outcome collinearity. We evaluate the stability of the concept importance scores generated by the different C-XAI methods when the concept-outcome distribution changes in the test data. We select the âłmicro-00M_micro-00 model and generate C-XAI explanations for the microscope concept on the test split of adulterated dataset variants micro-10D_micro-10 through micro-90D_micro-90. The spurious correlation between MELâmicroscope steadily increases in these test datasets (Figure 2b). C-XAI explanations should reflect the modelâs behaviour, not the test data statistics. Since âłmicro-00M_micro-00 was trained on data with no microscope artifacts, the ground-truth importance of microscope for this model is exactly zero. A reliable C-XAI method should therefore assign zero importance to microscope across all test variants, regardless of the artifactâs prevalence in the test data. Experiment 2b: false positives from concept-concept collinearity. Next, we evaluate whether C-XAI methods correctly assign zero importance to a concept that the model has never seen during training, even when the collinearity between this concept and other concepts change in the test data used for generating explanations. To test this, we insert a new synthetic timestamp artifact in k% of the microscope-containing images in the micro-40D_micro-40 dataset and generate new dataset variants micro-40+time-âkD_micro-40+time-k . From micro-40+time-10D_micro-40+time-10 through micro-40+time-90D_micro-40+time-90, the timestamp-microscope collinearity increases steadily, as shown in Figure 2d. We select the model âłmicro-40M_micro-40 which has learned to use the microscope artifact to some extent (Îâaccâ2% â 2\%) and generate C-XAI explanations for the timestamp concept on the test splits of micro-40+time-âkD_micro-40+time-k. Experiment 2c: responsiveness to an increasing degree of shortcut learning. Finally, we evaluate whether the C-XAI methods are responsive to increasing levels of shortcut learning. We apply all four methods to the 10 models âłmicro-00M_micro-00 through âłmicro-90M_micro-90, and compute the importance of the microscope concept at all layers. For consistency in comparison, C-XAI explanations are computed on the same test split of micro-10D_micro-10 where the synthetic microscope artifact is present at a low rate. A reliable C-XAI method should reflect the modelâs increasing reliance on the artifact by assigning ImpÂŻâ(microscope) Imp( microscope) that increases monotonically with increasing Îâacc . 3.6 Experiment 3: Neuroimaging applications Experiment 3 comprises two prediction tasks on UK Biobank brain MRI: binge-drinking classification (Experiment 3a) and brain-age regression (Experiment 3b). Data and preprocessing. We use T1-weighted structural MRI from the UK Biobank Sudlow et al. (2015), preprocessed to 96Ă114Ă9696Ă 114Ă 96 voxels with the standard UK Biobank pipeline (skull-stripping and non-linear registration to MNI-152 space) Alfaro-Almagro et al. (2021). Both tasks use five-fold stratified cross-validation (an 80%/20% train/validation split per fold, stratified by the outcome), and all reported metrics are averaged over the folds. The task-specific cohorts and splits are given with each experiment below. Concept set. As listed in Extended Data Table 3, both tasks are interpreted with 30 concept variables spanning demographics, cognitive tests, self-reported mental health, physical health, socioeconomic status, and MRI acquisition settings. After appending the respective outcome variable to the concept set, |âČ||C | becomes 31 variables. We standardise continuous variables and one-hot encode categorical variables. Extended Data Figure 10 shows the pairwise correlations between the concepts and the outcome variable. Several concepts are strongly intercorrelated, as is common in neuroimaging cohorts Alfaro-Almagro et al. (2021). C-XAI interpretation. Throughout the paper we index interpreted layers relative to the modelâs output, so LH_L is the prediction logit and LâkH_L-k the layer k steps before it. For each model we extract latent representations from several layers, spanning from the input layer (H1H_1), to the intermediary convolutional layers (e.g., HLâ16H_L-16 and HLâ11H_L-11 in the ResNet-18 model with L=20L=20), to the final feature-extractor layers (e.g., HLâ1H_L-1), plus the output prediction (e.g., HLH_L). Then we compute ICON importance and the normalised linear-probe importance on each (Equation 8) using the held-out test set of the respective prediction task. However, we do not compare against any CAV-based methods (TCAV Kim et al. (2018), signal-CAV Pahde et al. (2025a), RCAV Graziani et al. (2018)) here because they are incapable of handling categorical and continuous concepts uniformly. For example, TCAV is defined for categorical concepts and RCAV for continuous ones. Thus, they cannot produce importance scores comparable across our mixed 30-concept set. Furthermore, we have already reproduced the finding that CAV-based metrics are unstable Nicolson et al. (2025) in our skin-cancer experiment (Figure 3). Binge-drinking classification (Experiment 3a). The outcome variable is derived from the UK Biobankâs self-reported binge-drinking frequency field (Field ID 20416). To binarise the variable we use the same threshold as our reference study Rane et al. (2022), whereby participants with three or more binge episodes per month are categorised as the positive class and participants that report zero binge-drinking episodes are categorised as the controls. We drop participants with responses that are intermediate between these two extreme groups. From this pool of participants, we construct two subsamples with identical age and sex marginals that differ only in their sexâoutcome correlation. The first sample with n=6,255n=6,255 (5,009 train / 1,246 hold-out, 52% positive) retains the cohortâs natural correlations (Pearson râĄ(sex,outcome)â0.23r( sex,outcome)â 0.23) and the second one with n=6,309n=6,309 (5,039 train / 1,270 hold-out, 51% positive) is balanced by sex such that râĄ(sex,outcome)â0.00r( sex,outcome)â 0.00. We train a 3D ResNet-18 Hara et al. (2018), initialised from weights pretrained on human-actions video data and fully fine-tuned, on each subsample under identical hyperparameter settings: Adam (learning rate 1.1Ă10â51.1Ă 10^-5, weight decay 2.7Ă10â62.7Ă 10^-6, tuned with Optuna Akiba et al. (2019)), batch size 32, and up to 120 epochs with early stopping (patience 15). The natural-correlation model reaches 60±0.5%60± 0.5\% accuracy and the sex-balanced model only 53.3±0.5%53.3± 0.5\% (chance 50%50\%); because the two subsamples differ only in their sexâoutcome correlation, this gap isolates the predictive contribution of sex (Figure 5a shows the natural-correlation model). Additionally, to rule out architecture-specific effects, we repeat the prediction task also with the Simple Fully Convolutional Network (SFCN) Peng et al. (2021) architecture under the same training and tuning protocol and compare the ICON explanations for both model architectures (Extended Data Figure 11b). Brain-age prediction (Experiment 3b). Here the target is chronological age (regression). We sample n=10,000n=10,000 âhealthyâ participants (mean age 54.7±7.154.7± 7.1 years) for training by excluding anyone with a prior neurological or psychiatric diagnosis (ICD-10 chapters F and G), as the brain-age-gap paradigm requires Cole et al. (2019), and n=5,000n=5,000 participants (mean age 54.4±7.554.4± 7.5 years) as the test set drawn from the remaining 11,35411,354, which does include participants with neurological or psychiatric diagnoses. The subjects remaining in the 11,35411,354-participant pool after the test set is drawn are used in the independent validations described below, to generate the out-of-distribution and matched-control sets. We use the Simple Fully Convolutional Network (SFCN) which won the 2019 PAC brain-age challenge Peng et al. (2021), and train it with Adam (learning rate 1.5Ă10â31.5Ă 10^-3, weight decay 2.7Ă10â82.7Ă 10^-8; both selected with Optuna Akiba et al. (2019)), batch size 32, and a ReduceLROnPlateau schedule for up to 150 epochs with early stopping (patience 15) on the validation loss; the age target is standardised before fitting. Across five folds on the fixed test set the model reaches a mean absolute error of 2.89±0.022.89± 0.02 years and R2=0.750±0.004R^2=0.750± 0.004. To also visualise how concept importance evolves during the DNN optimisation, we additionally apply ICON to several training checkpoints of one of the five folds (Extended Data Figure 11a). Validating concept use in brain-age prediction. To check whether the concepts flagged by ICON or the linear probe are actually used by the brain-age model, we design two complementary validation procedures, both anchored to the age marginal of the fixed n=5,000n=5,000 test set for all five cross-validation folds. We take the union of the five concepts ranked most important (other than age) at the final hidden layer (Lâ1H_L-1) by ICON and by the probe. The concept importances are averaged across folds before ranking them, yielding eight unique concepts as listed in Table 1. The probe ranks Employment status as the most important concept after age and ICON ranks Acquisition date, whereas Snap reaction time and Systolic blood pressure appear in the top five of both. Validation 1: matched-resampling OOD test. For each of the eight selected concepts, we create an OOD test dataset. For each candidate concept we draw two equally sized sets from that remaining held-out pool: an OOD set in which the conceptâs correlation with age is removed, and a matched control in which it is left at its natural value (Extended Data Figure 12). We ensure that both the OOD and the matched control sets have the same marginal age distribution as the fixed n=5,000n=5,000 test set. We achieve this by binning age into 2-year bins and matching each binâs count, so they share the same age marginal and sample size. Within each bin the OOD set additionally balances subjects across the conceptâs categories (or global quantile bins, for continuous concepts), driving its correlation with age to zero. The only intended difference between the two sets is therefore the conceptâs correlation with age, so any error gap ÎâMAE=MAEOODâMAEcontrol =MAE_OOD-MAE_control isolates how much the modelâs accuracy depends on that correlation. We expect ÎâMAE>0 >0 if the model uses the concept as a proxy for age, whereas a false-positive concept should show ÎâMAEâ0 â 0. We set sampling thresholds to ensure the OOD and control sets are reasonably matched: (a) the residual concept-age correlation stays below 0.030.03, (b) age mean and variance match the reference test data to within 0.050.05 years and 1%1\%, and (c) the final sample size is nâ„200nâ„ 200. We could create OOD sets for six of the eight concepts with these thresholds, while two (Acquisition date and Employment status) proved to be too collinear with age to create a reasonable OOD test set. We assess each ÎâMAE per fold with a two-sample z-test approximation (SâE=Ï^â1/nOOD+1/ncontrolSE= Ï 1/n_OOD+1/n_control, Ï^â2.2 Ïâ 2.2 years from the reference hold-outâs dispersion of |y^âage|| y-age|) and call a concept significant only if it clears p<0.05p<0.05 in every fold, since the folds reuse the same test subjects and are therefore not independent. Validation 2: age-conditioned partial R2R^2. Because an OOD set is infeasible for two of the eight concepts, we add a secondary check for them by measuring the fraction of the predictionâs variance attributable to a concept ckc_k beyond age. Therefore, per fold, we fit two ordinary-least-squares models of the predicted age, f1:y^âŒagef_1: y and f2:y^âŒage+kf_2: y +c_k, and compute the partial ÎâR2=(Rf22âRf12)/(1âRf12) R^2=(R^2_f_2-R^2_f_1)/(1-R^2_f_1), the fraction of the age-independent variance in y y explained by kc_k, with significance from a nested-model F-test (Bonferroni-corrected across the two concepts). A near-zero ÎâR2 R^2 means no age-independent signal from kc_k is detectable in the prediction; it does not prove the model never uses the concept internally, and a non-zero ÎâR2 R^2 does not exclude the model relying on a higher-order correlate rather than kc_k itself. We therefore treat this test as complementary to the OOD test above, rather than confirmatory, and report effect sizes alongside significance. 4 Extended Data (Supplementary) 4.1 Literature review of concept-based explainability methods Concept-based explainability (C-XAI) methods quantify how important a human-understandable concept such as patient sex, a disease indicator, or an imaging artefact, is to a deep neural network (DNN), by analysing the networkâs internal representations. They have gained popularity due to their flexibility as they can interpret any trained DNN model in terms of concepts a domain expert chooses, when annotations are available. Furthermore, this can inform post-hoc model auditing and model steering features Poeta et al. (2023); Gupta and Narayanan (2024); Erogullari et al. (2026). The two dominant approaches are linear probes and Concept Activation Vectors (CAVs). We review each below and show that both inherit the same univariate decoding limitation that motivates ICON. Linear probes. A linear probe is a linear model trained on a DNN layerâs representation to predict a concept. The probeâs accuracy is then used as a measure of how strongly that concept is encoded in that representation Alain and Bengio (2017); Belinkov (2022). The central problem is that decodability does not imply use: a concept can be linearly decodable from a layer without the model ever relying on it to make predictions. Ravichander et al. Ravichander et al. (2021) show that probes recover concepts with high accuracy even when those concepts are irrelevant to the task, so probing accuracy alone cannot establish importance. Elazar et al. Elazar et al. (2021) make the same point causally: removing a property from the representation through an âamnesicâ intervention often leaves the modelâs predictions unchanged, confirming that decodability and behavioural relevance are distinct. Using probe accuracy as an importance score therefore produces false positives for concepts that are merely correlated with the representation or with the outcome. Concept Activation Vectors. CAV-based methods were introduced to close this specific gap between decodability versus use, by measuring not only if a concept is decodable, but also how sensitive the modelâs output is to it. A CAV is the direction in activation space separating concept from non-concept examples; TCAV Kim et al. (2018) then scores importance using the directional derivative of the output along this direction, and Regression Concept Vectors (RCAV) Graziani et al. (2018) extend the idea to continuous concepts. Because the score relies on the modelâs gradient, a concept the model does not use should receive a low importance score. In practice, however, CAV-based scores inherit two problems, the first of which is instability. The concept importance scores can shift substantially under small changes to the conceptâs training examples and to the choice of probing dataset Ramaswamy et al. (2023), and under adversarial perturbations of the inputs Brown and Kvinge (2023). When the same concept is measured at different adjacent layers of the same network, they often generate inconsistent importance scores Nicolson et al. (2025). The second problem, which CAVs share with probes, is that a conceptâs score absorbs the contribution of the concepts it is correlated with. Concept entanglement. A failure common to both probes and CAVs is concept entanglement: because a layerâs representation mixes many correlated concepts, an estimate for one concept can absorb the contribution of others that co-occur with it Nicolson et al. (2025); Raman et al. (2024). Erogullari et al. Erogullari et al. (2026) demonstrate this directly on the CelebA dataset, where âbeardâ and ânecktieâ co-occur almost exclusively in images of men: CAVs trained for the two concepts point in nearly the same (non-orthogonal) direction, so steering a generative model along the ânecktieâ direction also introduces correlated facial-hair concepts such as a moustache. The two concepts cannot be told apart from the representation alone, and any univariate score will attribute their shared variance to both. This is precisely why a concept the model never used can still be flagged as important whenever it is correlated with a concept (or with the outcome) that the model did use. Several refinements have been proposed: signal-based CAVs that recover more faithful concept directions by moving away from decoability Pahde et al. (2025a), orthogonalisation that separates correlated concept directions after training Erogullari et al. (2026), and usage recommendations and correlation-aware diagnostics for interpreting CAVs Nicolson et al. (2025); Raman et al. (2024). Each addresses an individual symptom such as direction noise, layer sensitivity, or pairwise entanglement, rather than the shared root cause. That root cause is the univariate decoding framing itself. Since probes and CAVs analyse each concept individually at a time, they cannot separate a conceptâs unique contribution from the contributions from all other correlated concepts and outcome. ICON addresses this directly by reframing C-XAI as a multivariate variance-decomposition problem, in which all concepts are evaluated jointly and shared variance is allocated explicitly rather than counted twice (see main text). Causal coefficients DNN Model Ground-truth importance ImpÂŻtrue(l) Imp^(l)_true (± std.dev.) True Signal ÎČld _l_d Confounders ÎČc1 _c_1 Confounders ÎČc2 _c_2 BAC (%) [min - max] Layer True Signal dl_d (%) Confounders c1c_1 (%) Confounders c2c_2 (%) Encoded in Image (%) Encoded in Outcome (%) Correlated Concept (%) Independent Concepts (%) 0.0000 0.6190 0.8473 73 Lâ2H_L-2 1 ± 1 11 ± 4 5 ± 2 84 ± 6 0 ± 0 0 ± 0 0 ± 0 [71â74] Lâ1H_L-1 1 ± 0 35 ± 12 18 ± 11 46 ± 18 0 ± 0 0 ± 0 0 ± 0 LH_L 0 ± 0 77 ± 4 21 ± 4 2 ± 1 0 ± 0 0 ± 0 0 ± 0 0.6190 0.6190 1.7346 74 Lâ2H_L-2 2 ± 1 6 ± 2 10 ± 5 82 ± 8 0 ± 0 0 ± 0 0 ± 0 [71â76] Lâ1H_L-1 5 ± 1 22 ± 8 35 ± 7 38 ± 15 0 ± 0 0 ± 0 0 ± 0 LH_L 8 ± 1 34 ± 6 56 ± 5 2 ± 0 0 ± 0 0 ± 0 0 ± 0 1.0986 0.7082 0.0000 77 Lâ2H_L-2 6 ± 3 13 ± 6 1 ± 1 80 ± 9 0 ± 0 0 ± 0 0 ± 0 [75â79] Lâ1H_L-1 21 ± 5 43 ± 10 1 ± 1 35 ± 13 0 ± 0 0 ± 0 0 ± 0 LH_L 24 ± 3 75 ± 3 0 ± 0 1 ± 1 0 ± 0 0 ± 0 0 ± 0 1.2657 0.0000 2.1972 77 Lâ2H_L-2 8 ± 3 1 ± 0 17 ± 6 74 ± 9 0 ± 0 0 ± 0 0 ± 0 [76â78] Lâ1H_L-1 20 ± 3 0 ± 0 48 ± 8 32 ± 10 0 ± 0 0 ± 0 0 ± 0 LH_L 28 ± 2 0 ± 0 71 ± 1 1 ± 0 0 ± 0 0 ± 0 0 ± 0 1.3863 0.2007 2.1972 78 Lâ2H_L-2 9 ± 3 1 ± 0 16 ± 5 74 ± 8 0 ± 0 0 ± 0 0 ± 0 [75â79] Lâ1H_L-1 21 ± 4 2 ± 0 44 ± 10 33 ± 14 0 ± 0 0 ± 0 0 ± 0 LH_L 33 ± 1 2 ± 0 64 ± 1 1 ± 0 0 ± 0 0 ± 0 0 ± 0 1.7346 0.0000 1.2657 76 Lâ2H_L-2 20 ± 7 0 ± 0 7 ± 2 72 ± 9 0 ± 0 0 ± 0 0 ± 0 [75â77] Lâ1H_L-1 47 ± 9 0 ± 0 16 ± 3 37 ± 12 0 ± 0 0 ± 0 0 ± 0 LH_L 73 ± 2 0 ± 0 26 ± 1 1 ± 0 0 ± 0 0 ± 0 0 ± 0 Table 2: Experiment 1: causal scenarios, model accuracy, and layer-wise ground-truth importance. The six data-generating settings used in Experiment 1, simulating causal scenarios from the data-generating graph (Figure 1b) that range from predominantly confounder-driven to predominantly signal-driven, ordered by ÎČd _l_d down the table. ÎČd _l_d is the coefficient of the unobservable true predictive signal on the outcome, while ÎČ1 _c_1 and ÎČ2 _c_2 are the coefficients of the two confounders on the outcome (Equation 9). For each scenario we train 1515 models (five layer-width variants Ă three random seeds) and report the mean balanced accuracy (BAC) on the test split, with the minimumâmaximum range across models in brackets. The remaining columns give the normalised ground-truth importance ImpÂŻtrue(l) Imp^(l)_true of every concept role as a percentage of the total importance across all 100100 concepts and the true signal, so each layer row sums to 100100. Because the ground truth differs across layers, we report one row per interpreted layer (Lâ2H_L-2, Lâ1H_L-1, LH_L). For readability we show the median across the 1515 models, but each model variation and seed is scored against its own ground truth in the experiments. Three concept roles âencoded in outcomeâ, âcorrelated conceptâ, and âindependent conceptâ, have no causal path into the image and therefore have zero ground-truth importance by construction in every scenario and at every layer. Also, note that in ToyBrains Rane et al. (2024a), the importance ordering of 1c_1 and 2c_2 need not follow the ordering of their ÎČ values as it depends on not just on the strength of the causal edge into the outcome ÎČ but also on its edge into an image attribute, its data type, and on how many states it has. Thus, we estimate the ground-truth empirically rather than reading it off the generating coefficients. Across all six scenarios, the shift in the ground-truth importance from the layer Lâ2H_L-2 to LH_L of the 1313 image-encoded concepts shows that the networks progressively discard image content irrelevant to the prediction. Figure 6: Sensitivity to concept-set size, sample size and layer width. Error from the ground truth (TVD, Equation 11) at each interpreted layer, for the same 9090 models and the same ground truth as Figure 1d. Lines are medians over the n=90n=90 models with 2525th to 7575th percentile bands. a, Varying the number of concepts supplied, at fixed n=500n=500. The concept pool is dominated by zero-importance concepts, so raising q mainly raises how many concepts a method must reject (for example 33 at q=5q=5, 2626 at q=30q=30 and 8585 at q=100q=100). b, Varying the number of test samples used to compute the explanation, at fixed q=30q=30. c, Varying the width plp_l of the interpreted layer, at fixed n=500n=500 and q=30q=30. The five points are the five width variants set for (Lâ2,Lâ1)(H_L-2,H_L-1) from (512,128)(512,128) through (8192,2048)(8192,2048). Every methodâs error varies modestly with width. ICON is the most accurate method at every width and at both layers. LH_L is omitted because pL=1p_L=1 in every variant. Figure 7: Un-normalised concept importance scores for Experiment 2 (compare with Figure 3). Same three analyses as Figure 3aâc, but plotting the raw importance scores ImpâĄ(ck)Imp(c_k) instead of the normalised ImpÂŻâ(ck) Imp(c_k). In Figure 3, each score is divided by the total importance assigned to all concepts and the outcome at the same layer (equation 8), so that layers with different dimensionalities become comparable and scores share a common scale across methods. Here we show the un-normalised outputs of each method directly. Colours denote C-XAI methods; line styles denote layers in the VGG-16 model with L layers. a, Specificity to conceptâoutcome collinearity: fixed model âłmicro-00M_micro-00 evaluated on test sets micro-âkD_micro-k with increasing microscopeâMEL collinearity. b, Specificity to conceptâconcept collinearity: fixed model âłmicro-40M_micro-40 evaluated on test sets micro-40+time-âkD_micro-40+time-k with increasing timestampâmicroscope collinearity, for the timestamp concept that the model has never seen during training. c, Responsiveness: models âłmicro-00M_micro-00 through âłmicro-90M_micro-90 evaluated on a fixed test split of micro-10D_micro-10 as shortcut reliance increases. The qualitative conclusions of Figure 3 hold in the un-normalised scores: ICON (green) remains near zero at all classifier-head layers in a and b and rises monotonically in c, while the three baselines show the same false-positive pattern reported in the main text. That is, all three assign a large non-zero score to a concept whose true importance is zero, and the linear probe tracks the test set correlations while the two CAV scores remain unstable. Scores from different methods are not directly comparable on this un-normalised scale (Methods 3.3). Figure 8: Composition of the ISIC 2019 dataset and the correlations among its outcome classes and artifact concepts. a, Histogram of the nine outcome classes (top) and the nine artifact concepts (bottom). The seven natural artifacts (blue bars) are band_aid, ruler, red_color, white_dots, skin_marker, reflection and piano_like_ruler. The two synthetic artifacts (green bars), microscope and timestamp, appear only in the adulterated variants; the counts shown are those of micro-40D_micro-40. b, Pearson correlations between every pair of the nine outcome classes and the nine artifact concepts, including the correlations of the artifact concepts with one another. The natural artifacts are also correlated with the outcome classes, for example red_color with NV (r=0.22r=0.22), ruler with MEL (r=0.18r=0.18) and reflection with BKL (r=0.16r=0.16). This is the background collinearity against which every C-XAI method in Figure 3 has to isolate the inserted artifact. Figure 9: ICON with the unexplained share versus normalised ICON for âłmicro-00M_micro-00 (left) and âłmicro-40M_micro-40 (right). The top row shows the raw ICON decomposition: a black segment at each layer represents the residual variance Ïl2Ï^2_l that is not linearly explained by any of the nine artifact concepts or nine outcome classes, while coloured segments represent variance attributed to each concept. The bottom row shows the same decomposition with each layerâs coloured segments rescaled to sum to one (the version used in Figure 4). The unexplained fraction shrinks from the early convolutional layers towards the prediction logits in both models, indicating that the supplied concept set accounts for a larger share of the latent variance in deeper layers. Rescaling the decompositions to a common scale makes per-concept proportions directly comparable across layers and across models, at the cost of hiding the absolute share of variance the concepts capture. Table 3: Concept set for the neuroimaging experiments. The concept variables supplied to ICON and the linear probe, grouped by category, with each variableâs type and UK Biobank cohort summary statistics. Both experiments draw on this shared pool of 3131 variables and supply 3030 of them as concepts. Each experiment sets aside the variable used to derive its own outcome: brain-age prediction sets aside chronological age, and binge-drinking classification sets aside frequency-of-alcohol-use. Statistics are computed over the union of the training and hold-out subjects for which concept annotations were supplied in the two experiments (n=17,713n=17,713 unique subjects; brain-age cohort n=15,000n=15,000, binge-drinking cohort n=6,255n=6,255, with 3,5423,542 subjects in both). Continuous variables are standardised to zero mean and unit variance; categorical and binary variables are one-hot encoded. Concept Category Type Statistics BingeDrinking Outcome Categorical Binge (52%) Non-binge (48%) Age Demographics Continuous 40.0 â 70.0 (ÎŒ=54.3) Handedness Demographics Categorical Right-handed (89%) Left-handed-or-both (11%) Sex Demographics Categorical Male (51%) Female (49%) DeprivationIndex Socioeconomic Status Continuous -6.3 â 9.2 (ÎŒ=-1.9) EmploymentStatus Socioeconomic Status Categorical Employed (72%) Retired (22%) Unemployed (6%) HouseholdIncome Socioeconomic Status Categorical modal category 30% of n=17396 BMI Physical Health Continuous 13.4 â 58.7 (ÎŒ=26.5) BloodPressureSystolic Physical Health Continuous 81.0 â 252.0 (ÎŒ=139.6) Cholesterol Physical Health Continuous 2.2 â 12.9 (ÎŒ=5.7) HasNeurologicalDisease Physical Health Categorical False (98%) True (2%) PhysicalActivitiesMET Physical Health Continuous 0.0 â 19278.0 (ÎŒ=2416.1) FluidIntelligence Cognitive Tests Continuous 0.0 â 13.0 (ÎŒ=6.7) SnapReactionTime Cognitive Tests Continuous 347.0 â 1684.0 (ÎŒ=589.0) TrailMakingTest Cognitive Tests Continuous 0.0 â 3337.0 (ÎŒ=546.4) DepressionEver Mental Health Categorical True (55%) False (45%) EverAddicted Mental Health Categorical False (95%) True (5%) EverHadAnxiety Mental Health Categorical False (73%) True (27%) FrequencyOfAlcoholUse Mental Health Continuous 0.0 â 5.0 (ÎŒ=3.2) FrequencyOfCannabisUse Mental Health Continuous 0.0 â 4.0 (ÎŒ=0.3) FrequencyOfTobaccoUse Mental Health Continuous 0.0 â 4.0 (ÎŒ=0.1) Happiness Mental Health Categorical modal category 45% of n=13208 HasPsychiatricDisorder Mental Health Categorical False (94%) True (6%) SelfHarm Mental Health Categorical False (84%) True (16%) TraumaChildhood Mental Health Categorical False (91%) True (9%) TraumaLifethreat Mental Health Categorical False (68%) True (32%) DateOfAcquisition MRI Scanner Settings Continuous 2014â2019 ImagingSite MRI Scanner Settings Categorical Cheadle (71%) Newcastle (15%) Reading (15%) ScannerTablePosition MRI Scanner Settings Continuous -1221.0 â -1020.0 (ÎŒ=-1056.7) ScannerXPosition MRI Scanner Settings Continuous -36.3 â 17.3 (ÎŒ=0.5) ScannerYPosition MRI Scanner Settings Continuous 52.0 â 96.0 (ÎŒ=65.4) ScannerZPosition MRI Scanner Settings Continuous -96.8 â 124.5 (ÎŒ=-26.4) Figure 10: Concept and outcome intercorrelations in the UK Biobank cohort. Pearson correlation matrix over all 3232 variables used across the two tasks, the 3131 variables listed in Extended Data Table 3 plus the binge-drinking outcome, computed over the n=17,713n=17,713 subjects. Each cell is coloured by the sign and magnitude of the pairwise Pearson correlation (diverging scale), and concepts are grouped along both axes into the six categories used throughout: demographics, socioeconomic status, physical health, cognitive tests, self-reported mental health, and MRI scanner-acquisition settings. The concepts are strongly intercorrelated: for example, age correlates with several socioeconomic, physical-health and cognitive variables (|r|>0.2|r|>0.2). The scanner-acquisition variables are also highly intercorrelated, with acquisition date and scanner table position reaching |r|â0.5|r|â 0.5 and so are also the self-reported mental-health items. Figure 11: ICON tracks how a representation forms over training and compares two architectures on a common scale. In both panels the bars are layer-wise ICON importance, ordered from an input layer at the bottom to the prediction logit LH_L at the top, and the black segment is the share of representation variance that none of the 3030 supplied concepts explains. a, The brain-age SFCN model at three stages of training, all from fold 1 of 5: at initialisation (validation MAE =5.90=5.90 years), at epoch 1313 (MAE =4.34=4.34 years) and at convergence at epoch 9797 (MAE =2.90=2.90 years). Scanner-position concepts dominate the early-layer representation near initialisation, and age comes to dominate the final and penultimate layers as the model converges, so these layers learn to extract the age signal and filter out the scanner information present in the input images. b, The binge-drinking task for a video-pretrained 3D ResNet-18 (left; fold 1 of 5, validation accuracy 59.4%59.4\%; 60.0±0.5%60.0± 0.5\% over five folds) and for an SFCN (right; fold 1 of 5, validation accuracy 61.3%61.3\%; 61.2±0.7%61.2± 0.7\% over five folds), both evaluated on the n=1,246n=1,246 hold-out participants. Because ICON expresses importance as a proportion of representation variance, the two architectures are directly comparable. Sex explains a comparably large share of the final-layer representation in both (â0.40â 0.40 for the ResNet-18 and â0.46â 0.46 for the SFCN), so changing the architecture and the pretraining does not substantially reduce the modelâs reliance on the sex confound at convergence. Figure 12: Matched resampling removes each conceptâs correlation with age while leaving the rest of the correlation structure intact. For each of the six concepts that admitted a valid out-of-distribution (OOD) set (Table 1), we show the absolute pairwise correlations among all validation concepts and the outcome, age, in the OOD test set (left) versus the matched random-control set (right). In every OOD set the target conceptâs correlation with age (bottom age row) is driven to â0â 0, whereas the control retains its natural value (for example, â0.4â 0.4 for systolic blood pressure and â0.3â 0.3 for sex and household income). All other correlations, including the strong link between the two scanner variables, acquisition date and scanner table position (â0.5â 0.5), are preserved across the two sets. Decorrelating the target concept from age is therefore the only intended difference between the OOD and control sets (Methods subsection 3.6). Data availability The ISIC 2019 Challenge dataset is publicly available at https://challenge.isic-archive.com/landing/2019/. UK Biobank data are available to approved researchers through the UK Biobank Access Management System (https://w.ukbiobank.ac.uk); this work was conducted under application number 33073. The ToyBrains simulator and the exact configuration files used to generate the datasets for Experiment 1 are archived at https://doi.org/10.5281/zenodo.14509513. Code availability The ICON implementation and the analysis code for all three experiments are available at https://github.com/RoshanRane/ICON_decomposition.git. Author contributions RPR: Conceptualization, Methodology, Investigation, Software, Visualization, Writing original draft, Review & editing. MS: Conceptualization, Methodology, Investigation, Review & editing. MP: Conceptualization, Methodology, Review & editing. MAS: Conceptualization, Investigation. NTS: Conceptualization. MD: Conceptualization, Methodology, Investigation. FP: Methodology, Review & editing. WS: Supervision, Review & editing. SG: Supervision, Methodology, Review & editing. KR: Supervision, Methodology, Investigation, Review & editing. All authors reviewed and approved the final manuscript. Competing interests The authors declare no competing interests. Acknowledgements Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) â 459422098; 402170461 (Losing and Regaining Control over Drug Intake; SFB 265); 414984028 (FONDA; SFB 1404) to KR; 565356445 (Validating Explainable AI in Clinical Neuroimaging), 586414057 (Diagnostic Fairness: Disentangling Data Quality, Confounding, and Algorithmic Bias in Biomedical Regression) to MAS. We gratefully acknowledge funding by GemeinnĂŒtzige Hertie Stiftung and Cluster of Excellence âMachine Learning â New Perspectives for Scienceâ. References Adebayo et al. (2022) J. Adebayo, M. Muelly, H. Abelson, and B. Kim Post hoc explanations may be ineffective for detecting unknown spurious correlation. In International Conference on Learning Representations (ICLR), Virtual Event. External Links: Link Cited by: ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Akiba et al. (2019) T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama Optuna: a next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, p. 2623â2631. Cited by: §3.6, §3.6. Alain and Bengio (2017) G. Alain and Y. Bengio Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations (ICLR), Workshop Track, External Links: Link Cited by: item 1, §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Alfaro-Almagro et al. (2021) F. Alfaro-Almagro, P. McCarthy, S. Afyouni, J. L. Andersson, M. Bastiani, K. L. Miller, T. E. Nichols, and S. M. Smith Confound modelling in uk biobank brain imaging. NeuroImage 224, p. 117002. Cited by: §1.3, §3.6, §3.6, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Bach et al. (2015) S. Bach, A. Binder, G. Montavon, F. Klauschen, K. MĂŒller, and W. Samek On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10 (7), p. e0130140. Cited by: ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Belinkov (2022) Y. Belinkov Probing classifiers: promises, shortcomings, and advances. Computational Linguistics 48 (1), p. 207â219. Cited by: §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Brown and Kvinge (2023) D. Brown and H. Kvinge Making corgis important for honeycomb classification: adversarial attacks on concept-based explainability tools. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), p. 620â627. Cited by: §1.2, §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Chen et al. (2020) T. Chen, S. Kornblith, M. Norouzi, and G. Hinton A simple framework for contrastive learning of visual representations. In International conference on machine learning, p. 1597â1607. Cited by: item 1. Clouvel et al. (2025) L. Clouvel, B. Iooss, V. Chabridon, M. I. Idrissi, and F. Robin An overview of variance-based importance measures in the linear regression context: comparative analyses and numerical tests. Socio-Environmental Systems Modelling 7 (18681). Cited by: ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Cole et al. (2019) J. H. Cole, R. E. Marioni, S. E. Harris, and I. J. Deary Brain age and other bodily âagesâ: implications for neuropsychiatry. Molecular psychiatry 24 (2), p. 266â281. Cited by: item 2, §3.6. CrabbĂ© and van der Schaar (2022) J. CrabbĂ© and M. van der Schaar Concept activation regions: a generalized framework for concept-based explanations. In Advances in Neural Information Processing Systems, Vol. 35, p. 2590â2607. Cited by: item 2. Dinga et al. (2020) R. Dinga, L. Schmaal, B. W. J. H. Penninx, D. J. Veltman, and A. F. Marquand Controlling for Effects of Confounding Variables on Machine Learning Predictions. bioRxiv, p. 2020.08.17.255034. Note: Preprint at https://doi.org/10.1101/2020.08.17.255034 External Links: Link, Document Cited by: ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Dreyer et al. (2024) M. Dreyer, F. Pahde, C. J. Anders, W. Samek, and S. Lapuschkin From hope to safety: unlearning biases of deep models via gradient penalization in latent space. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 21046â21054. Cited by: §1.2, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Eitel et al. (2021) F. Eitel, M. Schulz, M. Seiler, H. Walter, and K. Ritter Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research. Experimental Neurology 339, p. 113608. Cited by: §1.3. Elazar et al. (2021) Y. Elazar, S. Ravfogel, A. Jacovi, and Y. Goldberg Amnesic probing: behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics 9, p. 160â175. Cited by: §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Erogullari et al. (2026) E. Erogullari, S. Lapuschkin, W. Samek, and F. Pahde Post-hoc concept disentanglement: from correlated to isolated concept representations. In Explainable Artificial Intelligence. xAI 2025, Communications in Computer and Information Science, Vol. 2576, p. 68â89. External Links: Document, Link Cited by: §2, §4.1, §4.1, §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Franke and Gaser (2019) K. Franke and C. Gaser Ten years of brainage as a neuroimaging biomarker of brain aging: what insights have we gained?. Frontiers in neurology 10, p. 789. Cited by: item 2. Geirhos et al. (2020) R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann Shortcut learning in deep neural networks. Nature Machine Intelligence 2 (11), p. 665â673. Cited by: §3.5, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Graziani et al. (2018) M. Graziani, V. Andrearczyk, and H. MĂŒller Regression concept vectors for bidirectional explanations in histopathology. In Understanding and Interpreting Machine Learning in Medical Image Computing Applications: First International Workshops, MLCN 2018, DLF 2018, and iMIMIC 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16-20, 2018, Proceedings 1, p. 124â132. Cited by: §1.3, §3.6, §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Grömping (2015) U. Grömping Variable importance in regression models. Wiley interdisciplinary reviews: Computational statistics 7 (2), p. 137â152. Cited by: §2, item 6, item 7, item 8, §3.1, §3.3, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Gupta and Narayanan (2024) A. Gupta and P. J. Narayanan A survey on Concept-based Approaches For Model Improvement. arXiv. Note: Preprint at https://arxiv.org/abs/2403.14566 External Links: Link, Document Cited by: §4.1. Hara et al. (2018) K. Hara, H. Kataoka, and Y. Satoh Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 6546â6555. Cited by: §1.3, §3.6. HernĂĄndez-PĂ©rez et al. (2024) C. HernĂĄndez-PĂ©rez, M. Combalia, S. Podlipnik, N. C. Codella, V. Rotemberg, A. C. Halpern, O. Reiter, C. Carrera, A. Barreiro, B. Helba, et al. BCN20000: dermoscopic lesions in the wild. Scientific data 11 (1), p. 641. External Links: Document, Link Cited by: item 2, §1.2, §3.5. Hewitt and Liang (2019) J. Hewitt and P. Liang Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan (Eds.), Hong Kong, China, p. 2733â2743. External Links: Document, Link Cited by: item 1. Hyatt et al. (2020) C. S. Hyatt, M. M. Owens, M. L. Crowe, N. T. Carter, D. R. Lynam, and J. D. Miller The quandary of covarying: a brief review and empirical examination of covariate use in structural neuroimaging studies on psychological variables. NeuroImage 205, p. 116225. External Links: ISSN 1053-8119, Document, Link Cited by: §1.3, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Kim et al. (2018) B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, and R. Sayres Interpretability beyond feature attribution: quantitative testing with concept activation vectors (tcav). In International conference on machine learning, p. 2668â2677. External Links: Link Cited by: item 2, §3.5, §3.6, §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Kutner et al. (2005) M. H. Kutner, C. J. Nachtsheim, J. Neter, and W. Li Applied linear statistical models. 5th edition, McGraw-Hill/Irwin, New York. Cited by: §2, §3.1, §3.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Li et al. (2024) M. Li, H. Jin, R. Huang, Z. Xu, D. Lian, Z. Lin, D. Zhang, and X. Wang Evaluating readability and faithfulness of concept-based explanations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 607â625. External Links: Document, Link Cited by: ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Nicolson et al. (2025) A. Nicolson, L. Schut, A. Noble, and Y. Gal Explaining explainability: recommendations for effective use of concept activation vectors. Transactions on Machine Learning Research. Cited by: §1.2, §3.6, §4.1, §4.1, §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Pahde et al. (2023) F. Pahde, M. Dreyer, W. Samek, and S. Lapuschkin Reveal to revise: an explainable ai life cycle for iterative bias correction of deep models. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 596â606. Cited by: ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Pahde et al. (2025a) F. Pahde, M. Dreyer, L. Weber, M. Weckbecker, C. J. Anders, T. Wiegand, W. Samek, and S. Lapuschkin Navigating neural space: revisiting concept activation vectors to overcome directional divergence. In International Conference on Learning Representations (ICLR), Cited by: §1.2, item 2, item 3, §3.5, §3.6, §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Pahde et al. (2025b) F. Pahde, T. Wiegand, S. Lapuschkin, and W. Samek Ensuring medical ai safety: interpretability-driven detection and mitigation of spurious model behavior and associated data. Machine learning 114 (9), p. 206. Cited by: §1.2, item 2, §3.5. Pearl et al. (2016) J. Pearl, M. Glymour, and N. P. Jewell Causal inference in statistics: a primer. John Wiley & Sons. Cited by: §3.4, §3.4. Peng et al. (2021) H. Peng, W. Gong, C. F. Beckmann, A. Vedaldi, and S. M. Smith Accurate brain age prediction with lightweight deep neural networks. Medical Image Analysis 68, p. 101871. External Links: Document Cited by: §1.3, §3.6, §3.6. Poeta et al. (2023) E. Poeta, G. Ciravegna, E. Pastor, T. Cerquitelli, and E. Baralis Concept-based explainable artificial intelligence: a survey. ACM Computing Surveys. External Links: Link, Document Cited by: §1.2, item 2, §3.3, §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Raman et al. (2024) N. J. Raman, M. Espinosa Zarlenga, and M. Jamnik Understanding inter-concept relationships in concept-based models. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 42009â42025. External Links: Link Cited by: §4.1, §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Ramaswamy et al. (2023) V. V. Ramaswamy, S. S. Kim, R. Fong, and O. Russakovsky Overlooked factors in concept-based explanations: dataset choice, concept learnability, and human capability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 10932â10941. Cited by: §1.2, §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Ramsundar et al. (2019) B. Ramsundar, P. Eastman, P. Walters, and V. Pande Deep learning for the life sciences: applying deep learning to genomics, microscopy, drug discovery, and more. OâReilly Media. Cited by: §1.3, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Rane et al. (2024a) Toybrains simulation dataset External Links: Document, Link Cited by: item 1, §1.1, §3.4, Table 2, Table 2. Rane et al. (2022) R. P. Rane, E. F. de Man, J. Kim, K. Görgen, M. Tschorn, M. A. Rapp, T. Banaschewski, A. L. Bokde, S. Desrivieres, H. Flor, et al. Structural differences in adolescent brains can predict alcohol misuse. Elife 11, p. e77545. Cited by: item 1, §3.6, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Rane et al. (2024b) R. P. Rane, J. Kim, A. Umesha, D. Stark, M. Schulz, and K. Ritter DeepRepViz: identifying potential confounders in deep learning model predictions. In International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 186â196. Cited by: ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Rane et al. (2023) R. P. Rane, M. P. M. Musial, A. Beck, M. Rapp, F. Schlagenhauf, T. Banaschewski, A. L. Bokde, M. P. Martinot, E. Artiges, F. Nees, et al. Uncontrolled eating and sensation-seeking partially explain the prediction of future binge drinking from adolescent brain structure. NeuroImage: Clinical 40, p. 103520. Cited by: item 1. Ravichander et al. (2021) A. Ravichander, Y. Belinkov, and E. Hovy Probing the probing paradigm: does probing accuracy entail task relevance?. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, p. 3363â3377. External Links: Document, Link Cited by: §4.1, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Selvaraju et al. (2017) R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra Grad-cam: visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), p. 618â626. Cited by: ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Shapley (1953) L. S. Shapley A value for n-person games. In Contributions to the Theory of Games I, p. 307â317. Cited by: item 8. Simnacher et al. (2026) M. Simnacher, X. Xu, H. Park, C. Lippert, and S. Greven Deep nonparametric conditional independence tests for images. Journal of Machine Learning Research 27 (96), p. 1â73. Cited by: §2. Simonyan and Zisserman (2015) K. Simonyan and A. Zisserman Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), Cited by: §3.5. Spisak (2022) T. Spisak Statistical quantification of confounding bias in machine learning models. GigaScience 11, p. giac082 (en). External Links: ISSN 2047-217X, Link, Document Cited by: §2. Sudlow et al. (2015) C. Sudlow, J. Gallacher, N. Allen, V. Beral, P. Burton, J. Danesh, P. Downey, P. Elliott, J. Green, M. Landray, et al. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS medicine 12 (3), p. e1001779. Cited by: item 1, §3.6. Thibeau-Sutre et al. (2022) E. Thibeau-Sutre, B. Couvy-Duchesne, D. Dormont, O. Colliot, and N. Burgos MRI field strength predicts alzheimerâs disease: a case example of bias in the adni data set. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), p. 1â4. Cited by: ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Wegelin (2000) J. A. Wegelin A survey of partial least squares (pls) methods, with emphasis on the two-block case. Technical report Department of Statistics, University of Washington. Cited by: §3.1. Winkler et al. (2019) J. K. Winkler, C. Fink, F. Toberer, A. Enk, T. Deinlein, R. Hofmann-Wellenhof, L. Thomas, A. Lallas, A. Blum, W. Stolz, et al. Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition. JAMA dermatology 155 (10), p. 1135â1141. Cited by: §1.2, ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Wold (1966) H. Wold Estimation of principal components and related models by iterative least squares. Multivariate analysis, p. 391â420. Cited by: ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing. Wold et al. (2001) S. Wold, M. Sjöström, and L. Eriksson PLS-regression: a basic tool of chemometrics. Chemometrics and intelligent laboratory systems 58 (2), p. 109â130. Cited by: §2, §3.1. Zhang et al. (2025) H. Zhang, X. Wang, C. Li, X. Ao, and Q. He Controlling large language models through concept activation vectors. Proceedings of the AAAI Conference on Artificial Intelligence 39 (24), p. 25851â25859. External Links: Document, Link Cited by: §2, item 2.