Paper deep dive
Local Additive Feature Attribution: A Mathematical Taxonomy and Reporting Checklist
Rebecca Afriyie Sarpong, Daniel Commey
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/17/2026, 9:24:50 AM
Summary
This survey paper introduces a unified mathematical taxonomy for local additive feature attribution methods in explainable AI. It organizes prominent methods (SHAP, Integrated Gradients, LIME, Grad-CAM, LRP, DeepLIFT) around five core specification choices: value function, reference, path, perturbation distribution, and conservation rule. The authors compare these methods via an axiom-by-method matrix, link common failure modes to underlying mathematical assumptions, and propose a ten-item reporting checklist to ensure transparent and reproducible attribution reporting.
Entities (15)
Relation Signals (11)
LIME â uses â Perturbation Distribution
confidence 96% ¡ Perturbation-based methods do not integrate but instead evaluate f at masked inputs. ... Perturbation distribution that defines local linearity
LRP â uses â Conservation Rule
confidence 96% ¡ Conservation rule: What quantity is propagated? LRP, DeepLIFT
DeepLIFT â uses â Reference
confidence 96% ¡ Reference: Compared to what? IG, DeepLIFT, GradientSHAP
SHAP â uses â Value Function
confidence 96% ¡ In Shapley-based attribution, the value function is often the most consequential modelling choice...
Integrated Gradients â uses â Path
confidence 96% ¡ Path-based methods integrate gradients along a continuous curve in input space. The canonical choice is the straight line... used by Integrated Gradients.
Local Additive Feature Attribution â categorizes â SHAP
confidence 95% ¡ This survey proposes a common framework for local additive feature attribution. It organizes Shapley, path-based, gradient/backpropagation, perturbation, and CAM-style methods around five specification choices...
Local Additive Feature Attribution â categorizes â Integrated Gradients
confidence 95% ¡ It organizes Shapley, path-based, gradient/backpropagation, perturbation, and CAM-style methods around five specification choices...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Feature-attribution methods are central to explainable artificial intelligence. Their assumptions are expressed in several mathematical languages: cooperative-game values, path integrals, gradient operators, perturbation distributions, and backpropagation rules. This survey proposes a common framework for local additive feature attribution. It organizes Shapley, path-based, gradient/backpropagation, perturbation, and CAM-style methods around five specification choices: value function, reference, path, perturbation distribution, and conservation rule. It then compares these methods through an axiom-by-method matrix and links common failure modes, including baseline sensitivity, off-manifold perturbations, sanity-check failures, adversarial manipulation, and method disagreement, to the assumptions that produce them. Finally, the survey proposes a ten-item reporting checklist for studies that use local additive attributions. The central message is that attribution results are meaningful only relative to the mathematical assumptions under which they are defined, and that those assumptions should be reported.
Tags
Links
- Source: https://arxiv.org/abs/2607.14271v1
- Canonical: https://arxiv.org/abs/2607.14271v1
Trouble viewing inline? Open PDF directly â
Full Text
191,192 characters extracted from source content.
Expand or collapse full text
Local Additive Feature Attribution: A Mathematical Taxonomy and Reporting Checklist Rebecca Afriyie Sarpong and Daniel Commey Abstract Feature-attribution methods are central to explainable artificial intelligence. Their assumptions are expressed in several mathematical languages: cooperative-game values, path integrals, gradient operators, perturbation distributions, and backpropagation rules. This survey proposes a common framework for local additive feature attribution. It organizes Shapley, path-based, gradient/backpropagation, perturbation, and CAM-style methods around five specification choices: value function, reference, path, perturbation distribution, and conservation rule. It then compares these methods through an axiom-by-method matrix and links common failure modes, including baseline sensitivity, off-manifold perturbations, sanity-check failures, adversarial manipulation, and method disagreement, to the assumptions that produce them. Finally, the survey proposes a ten-item reporting checklist for studies that use local additive attributions. The central message is that attribution results are meaningful only relative to the mathematical assumptions under which they are defined, and that those assumptions should be reported. I Introduction Feature attribution has become one of the dominant interfaces between complex predictive models and their human users. Faced with a model that maps a d-dimensional input xââdx ^d to a prediction fâ(x)ââf(x) , a user asks: which features mattered, and by how much? A feature-attribution method answers with a vector Ďâ(f,x)ââd Ď(f,x) ^d that distributes credit (or blame) for the prediction among the input features. Over the last decade, dozens of such methods have been proposed, including Integrated Gradients [93], SHAP [62], LIME [73], Grad-CAM [80], LRP [9], DeepLIFT [83], and many others, each motivated by a different intuition about what an explanation should be. The rapid growth of attribution methods has not been matched by comparable clarity about their assumptions. Two methods applied to the same prediction frequently produce explanations that agree only on coarse features and disagree, sometimes sharply, on their relative ordering. Krishna et al. [55] documented this phenomenon at scale across six attribution methods and four tasks, the median Spearman rank correlation between explanations of the same prediction was below 0.50.5. They termed it the disagreement problem. Bilodeau et al. [15] established a complementary negative result: no feature-attribution method can simultaneously satisfy a small set of reasonable desiderata, so disagreement among methods is mathematically unavoidable. Earlier critiques along similar lines include [4, 54]. Empirical evaluations on the same benchmarks reach incompatible conclusions about which method is most âfaithfulâ [94, 45]. And it is now well documented that adversarial manipulations can produce arbitrary attribution maps with negligible changes to the prediction [30, 39, 86]. These pathologies often trace to implicit mathematical choices that different methods make and that users rarely see. This survey uses those mathematical choices as its organizing principle and compares feature-attribution methods through their underlying mathematical objects: the value function v that defines âfeature presenceâ, the baseline or reference xâ˛x against which contributions are measured, the path Îł along which integration occurs, the perturbation distribution that defines local linearity, and the conservation rule that distributes a quantity through a network. Many important disagreements and failure modes can be traced to differences in one or more of these objects. Figure 1 summarizes how this framing organizes the paper. Model f, input x, scalar output fcâ(x)f_c(x)Attribution method Ďâ(f,x,xâ˛)ââd Ď(f,x,x ) ^d Hidden specification choices value function v â ¡ reference xâ˛x â ¡ path Îł perturbation distribution p_ z â ¡ conservation rule Axioms satisfied (Sec. I, Tab. IX) Failure modes (Sec. X) Reporting items R1âR10 (Sec. XII) determined by Figure 1: Attribution methods as specifications of hidden mathematical choices. A local additive attribution is determined by five specification choices (value function, reference, path, perturbation distribution, conservation rule); these choices fix which axioms the method satisfies and which failure modes it is exposed to, and the proposed reporting checklist asks studies to state them. TABLE I: Five mathematical choices used as the organizing frame for the survey. Object Question answered Examples affected Value function What does feature absence mean? SHAP, KernelSHAP, TreeSHAP Reference Compared to what? IG, DeepLIFT, GradientSHAP Path Along what trajectory? IG, Guided IG, Blur IG Perturbation distribution Which neighbourhood is local? LIME, occlusion, RISE Conservation rule What quantity is propagated? LRP, DeepLIFT I-A Why Axioms? A second organizing principle of this survey is the use of axioms as the primary tool of comparison. Axioms are precise properties an attribution method may or may not satisfy: completeness (the attributions sum to the prediction-minus-baseline), implementation invariance (functionally equivalent models receive identical attributions), sensitivity (a feature that changes the prediction must receive nonzero attribution), symmetry, dummy, and others. We adopt the axiomatic perspective for three reasons: 1. Axioms are model-agnostic. A claim like âIntegrated Gradients satisfies completenessâ is a theorem about the method, not a hypothesis about a particular dataset or architecture. This makes axioms the right invariants for a survey: they remain true across benchmarks [93, 63]. 2. Axioms expose disagreement at its source. When two methods produce different explanations, the question âwhich is correct?â is usually ill-posed; it amounts to asking which set of axioms one prefers [92, 56]. The function-approximation perspective of Han et al. [43] and the impossibility result of Bilodeau et al. [15] make the trade-off explicit: choosing a method is choosing which axioms to retain. Making the axioms explicit converts an unresolvable empirical dispute into a modeling decision. 3. Axioms enable equivalence and reduction theorems. Several apparently distinct methods can be shown to coincide under appropriate axiomatizations. KernelSHAP, exact Shapley, and certain forms of weighted linear regression compute the same quantity [62, 21]. DeepLIFT with the Rescale rule converges to Integrated Gradients in the limit of small input increments [6]. These equivalences are invisible from a heatmap-comparison perspective; they appear only through the axioms. I-B Attribution vs. Interpretability, Explanation, and Causality The terms interpretability, explanation, and attribution are often used interchangeably in the XAI literature, but the underlying objects are mathematically distinct. Interpretability is a property of a model, defined here as the extent to which a human can predict, audit, or modify its behaviour from its structure alone [60, 31]. Explanation is a broader category that includes example-based, counterfactual, concept-based, and rule-based justifications. Attribution is the narrow problem of decomposing a single prediction into per-feature contributions, typically as a real-valued vector. This survey concerns the third object: local, additive, post-hoc feature attribution for differentiable and tree-structured models. We further distinguish attribution from causal feature analysis. A causal feature effect asks how the prediction would change under an external intervention on a feature, in the sense of Pearlâs do-calculus. Most attribution methods, including all gradient-based methods, compute associational quantities relative to a chosen reference distribution. They become causal only under assumptions about the data-generating process that are rarely stated and even more rarely satisfied [49, 37]. Conflating the two is a major source of misinterpretation in high-stakes domains, and we return to it in Section X. I-C Scope Included. Local, post-hoc feature attribution for predictive models. This covers Shapley-value methods, path-integral methods, gradient and backpropagation methods, perturbation-based and surrogate methods, and CAM-style visual attribution insofar as it is derived from gradients. We treat attribution as a mathematical object and devote substantial space to axioms, value functions, baselines, paths, and conservation rules. We include the evaluation theory of faithfulness, infidelity, sanity checks, and ROAR-style retraining that has emerged in parallel with the methods themselves. Excluded or briefly treated. Inherently interpretable models (linear models, decision lists, rule sets), global rule extraction, counterfactual explanations, example-based explanations, mechanistic interpretability, concept-bottleneck models, and visualization tools without attribution semantics. We comment on the boundary with concept methods (TCAV [53], network dissection [14]) where useful, but treat them as a complementary research programme outside the attribution methods surveyed here. I-D Position Relative to Prior Surveys Several broad XAI surveys preceded this one, including the foundational surveys of Guidotti et al. [42], Adadi and Berrada [3], Gilpin et al. [41], the book-length treatment of Molnar [65], the global-interpretation survey of Saleem et al. [77], the gradient-method technical review of Wang et al. [98], the additive feature-attribution review of Cremades et al. [23], the Shapley-specific survey of Li et al. [58], and the systematic evaluation review of Nauta et al. [69]. Table I compares this paper against those references along eight dimensions. The present survey differs from this literature in two respects: 1. Existing surveys have treated several of these families in depth, but the cross-family axiom structure remains fragmented. This article jointly cross-references Shapley, path-based, gradient/backpropagation, and perturbation methods through a single axiom-by-method matrix. Existing surveys are either family-specific (Shapley [58], gradient [98], additive [23]) or breadth-prioritised [42, 3, 41]. 2. The assumption sensitivity of attribution methods has not yet been consolidated into a reporting checklist for papers that use local additive attributions. This paper proposes such a checklist in Section XII and maps it to the analytic corpus in Appendix B. Comparison criteria for Table I. We assigned â when the surveyed reference dedicates at least one section, subsection, or comparable structural unit to the column dimension; â when the dimension is treated in fewer than two paragraphs, in an appendix, or only as part of a broader framing; and blank when the dimension is not addressed. The table is a scope comparison, not a bibliometric ranking. Ambiguous cases are scored in favour of the prior survey, and the interpretation of each mark is recorded in the supplementary scoring sheet. TABLE I: Position of this survey against prior XAI/attribution surveys. â= covered substantively; â = brief or partial; blank = not covered. âShap.â Shapley methods; âIGâ path methods; âCAMâ Grad-CAM family; âAxiom matrixâ an explicit axiom-by-method cross reference; âVF tax.â value-function taxonomy; âPath/baseline tax.â baseline and path taxonomy; âEval/Failureâ formal evaluation and failure-mode treatment; âChecklistâ reporting checklist as a deliverable. The final row records the intended scope of the present work, not an independent quality ranking. Survey Shap. IG CAM Axiom matrix VF tax. Path/baseline tax. Eval/Failure Checklist Guidotti et al. 2018 [42] â â â Adadi & Berrada 2018 [3] â â â â Gilpin et al. 2018 [41] â â â Molnar 2022 [65] â â â â â â Linardatos et al. 2021 [59] â â â â Saleem et al. 2022 [77] â â Nauta et al. 2023 [69] â â â â Li et al. 2024 [58] â â â Wang et al. 2024 [98] â â â â Cremades et al. 2024 [23] â â â â â This work â â â â â â â â I-E Survey Methodology This paper is a structured narrative survey and taxonomy-driven review. The search aimed to assemble the methodological lineage needed to compare local attribution operators mathematically; estimating the size of the XAI literature or achieving exhaustive bibliometric coverage was out of scope. The analytic corpus contains 105 unique cited works spanning method-defining papers, axiomatic characterizations, evaluation studies, failure-mode analyses, boundary cases, and prior surveys. Papers were selected for their role in the taxonomy: defining an attribution operator, stating an axiom or reduction, evaluating attribution behaviour, documenting a failure mode, or clarifying the boundary between attribution and adjacent forms of explanation. Databases. Google Scholar, Semantic Scholar (S2-API), the ACL Anthology, and the proceedings indices of NeurIPS, ICML, ICLR, AAAI, IJCAI, ECCV, ICCV, CVPR, ACL, EMNLP, NAACL, and KDD. Time range. Searches were conducted between January and April 2025. The planned publication window ran from 1953 (Shapleyâs n-person-games paper) through the end of 2024. We also included the directly relevant 2025 axiomatic characterization of Integrated Gradients by Lundstrom and Razaviyayn [63], identified during final verification. Query strings. We ran the following queries verbatim on each database; the conjunctions were taken as Boolean AND, the disjunctions as Boolean OR. ⢠("feature attribution" OR "Shapley value" OR "Integrated Gradients" OR LRP OR DeepLIFT OR Grad-CAM OR SHAP OR LIME) AND (axiom OR axiomatic OR completeness OR consistency) ⢠("Aumann-Shapley" OR Banzhaf OR "Owen value" OR "quantitative input influence" OR "game theory") AND ("feature attribution" OR "individual prediction" OR explanation) ⢠("attribution" OR "saliency") AND (faithfulness OR infidelity OR "sanity check" OR ROAR OR insertion-deletion) ⢠("attribution" OR "explanation") AND (adversarial OR fragile OR manipulation OR Lipschitz) Selection process. Candidate records were deduplicated by title, DOI, and arXiv identifier where available. Papers were retained when they satisfied at least one of the following roles: (i) introduced a method included in the taxonomy, (i) supplied an axiomatic characterization or reduction used in the comparison matrix, (i) proposed an evaluation metric or benchmark used in Section IX, (iv) documented a failure mode used in Section X, or (v) provided a prior survey against which the present article is positioned. Historical sources were retained when they are needed for the mathematical genealogy of Shapley values, cooperative-game attribution, or early saliency methods. TABLE I: Review corpus summary. Role counts are non-exclusive because one cited paper may introduce a method, state an axiom, and provide an evaluation result. Role in analytic corpus Included papers Method-introducing papers 56 Axiomatic characterization papers 26 Evaluation / benchmark papers 20 Failure-mode papers 15 Prior surveys 10 Boundary-case papers 15 Total unique cited papers in analytic corpus 105 Boundary and exclusion criteria. The primary corpus is restricted to local additive attribution for predictive models. Global-interpretability methods, counterfactual and example-based explanations, concept-based explanations, and mechanistic interpretability are treated only when they clarify a boundary case for the taxonomy. Duplicate versions of the same work were represented by the most complete or most widely cited version. Corpus construction. The corpus is organized by analytic role. This choice is appropriate for a taxonomy whose unit of comparison is the attribution operator and its mathematical assumptions. Reproducibility is supported through the query strings, inclusion roles, boundary criteria, and the final analytic corpus underlying the tables. The corpus emphasizes English-language sources from major machine-learning, computer-vision, natural-language-processing, and AI venues, together with foundational mathematical sources required for the Shapley and axiomatic lineage. I-F Contributions and Roadmap The paper contributes three concrete artifacts: a taxonomy, an axiom matrix, and a reporting checklist. 1. A unified mathematical taxonomy (Sections IâVII). Section I fixes a common notation for f, x, xâ˛x , the coalition algebra over N=1,âŚ,dN=\1,âŚ,d\, value functions v, masks z, and paths Îłâ(Îą)Îł(Îą). Section I catalogues the axioms under which different methods are characterized. Sections IVâVII survey the four method families in this common frame. 2. An axiom-by-method matrix and failure-mode formalization (Sections VIIIâX). Section VIII presents the main comparison table of the paper, the axiom-by-method matrix (Table IX), together with the complexity landscape and several known reductions between methods (KernelSHAP and exact Shapley in expectation [62, 21]; DeepLIFT and Integrated Gradients in the small-increment limit [6]; Grad-CAM and class-conditional gradient projections [80]; LRP-Ͼξ and gradient-Ă-input on bias-free ReLU networks [6]). Section X recasts the most-cited critiques of attribution methods [4, 54, 39, 86, 56] as mathematical problems tied to explicit modelling choices. 3. A proposed reporting checklist (Section XII). Feature-attribution methods encode modelling choices. Section XII expresses this point as a ten-item checklist for papers that report or rely on attribution results. The remainder of the paper is organized as follows. Section I fixes notation. Section I states the axioms. Sections IVâVII survey the four method families. Section VIII presents the unifying comparison framework. Section IX reviews evaluation theory. Section X analyzes failure modes. Section XI surveys applications across model families. Section XII presents the proposed reporting checklist. Section XIII states the scope boundaries of the survey. Section XIV closes with open problems and a research agenda. I-G Central Claim The argument of the paper is summarized by the following claim: There is no assumption-free feature-attribution method. Every local additive attribution method defines feature importance through choices about value functions, references, paths, perturbation distributions, or conservation rules. Trustworthy use therefore requires reporting a heatmap or ranking together with the assumptions under which the attribution was computed and interpreted. The checklist supports reproducible reporting of the assumptions required to compute and interpret an attribution. The remainder of the paper develops this claim formally and shows how common failure cases arise when these assumptions are left implicit. I Problem Formulation and Notation This section fixes a single notation used throughout the paper. Every method in Sections IVâVII is described in the symbols introduced here, making differences between methods visible as differences in specific mathematical objects across papers with otherwise distinct notation. I-A Model and Input Space We consider a predictive model f:ââ,f , where ââdX ^d is the input space and d is the number of input features. For multi-class problems, f is the scalar logit or pre-softmax score associated with a target class c; when the dependence on c matters, we write fcf_c. We make no structural assumption on f: it may be a deep network, a tree ensemble, or any other function from X to âR. Where differentiability is required, we will say so explicitly. The data distribution on X is denoted pXp_X, with XâźpXX p_X a random input. An individual input is written x=(x1,âŚ,xd)ââdx=(x_1,âŚ,x_d) ^d, with the i-th coordinate xix_i. I-B Baselines and References Every additive attribution method requires, implicitly or explicitly, a reference point against which the prediction fâ(x)f(x) is compared. We denote a single reference point as xâ˛ââdx ^d and a reference distribution as xâ˛D_x . Common choices include ⢠a fixed point such as the all-zeros vector xâ˛=x = 0, used in many gradient implementations; ⢠the population mean xâ˛=â[X]x =E[X]; ⢠a sample from the training distribution, xâ˛âźpXx p_X, giving an expected baseline [33]; ⢠a structured reference (e.g., a blurred image, a paraphrase of a sentence, an isoelectric protein sequence) appropriate to the input modality. We use Îâfâ(x;xâ˛)âfâ(x)âfâ(xâ˛) f(x;x ) f(x)-f(x ) for the prediction-minus-baseline difference, which most additive methods target as the quantity to decompose. I-C Coalitions and Value Functions Let N=1,2,âŚ,dN=\1,2,âŚ,d\ index the features. A coalition is a subset SâNS N of features deemed âpresentâ; its complement SÂŻ=NâS S=N S is the set of âabsentâ features. The number of features in S is |S||S|, and |N|=d|N|=d. A value function is a set function v:2Nââ,SâŚvâ(S),v 2^N , S v(S), intended to represent âthe prediction when only features in S are observedâ. In Shapley-based attribution, the value function is often the most consequential modelling choice, and the ambiguity in defining it is the source of much downstream disagreement. Three canonical choices appear repeatedly: Marginal (interventional) value. Replace absent features with their reference values: vxâ˛intâ(S)=xâ˛âźxâ˛â[fâ(xS,xSÂŻâ˛)],v^int_x (S)=E_x _x \! [f\! (x_S,x _ S ) ], (1) where (xS,xSÂŻâ˛)(x_S,x _ S) denotes the vector with feature i set to xix_i if iâSiâ S and to xiâ˛x _i otherwise. This corresponds to a hard intervention. Conditional value. Condition on observed features: vcondâ(S)=â[fâ(X)|XS=xS].v^cond(S)=E\! [f(X)\, |\,X_S=x_S ]. (2) This restricts to the data manifold but requires estimating d-dimensional conditional expectations [1, 37]. Single-reference value. The deterministic limit of the interventional value with a fixed baseline: vxâ˛refâ(S)=fâ(xS,xSÂŻâ˛).v^ref_x (S)=f\! (x_S,x _ S ). (3) The interventional and conditional value functions agree only when features are mutually independent under pXp_X. In all other cases they yield different Shapley values [49, 92, 64]. I-D Paths and Perturbations Path-based methods integrate gradients along a continuous curve in input space. A path from a baseline xâ˛x to an input x is a differentiable map Îł:[0,1]ââd,Îłâ(0)=xâ˛,Îłâ(1)=x.Îł [0,1] ^d, Îł(0)=x , Îł(1)=x. The canonical choice is the straight line Îłâ(Îą)=xâ˛+Îąâ(xâxâ˛)Îł(Îą)=x +Îą(x-x ), used by Integrated Gradients. Alternatives include adaptive paths [52], paths through image scale space [100], and paths defined by a generative model [17]. Perturbation-based methods do not integrate but instead evaluate f at masked inputs. A mask is a vector â[0,1]d zâ[0,1]^d that interpolates between the input and a reference; we use âx+(1â)âxⲠz x+(1- z) x to denote a masked input. When â0,1d zâ\0,1\^d the mask is binary, and we identify z with the coalition S=i:i=1S=\i: z_i=1\. I-E Attribution Vectors and Explanation Surrogates An attribution vector for the prediction fâ(x)f(x) relative to a baseline xâ˛x is a vector Ďâ(f,x,xâ˛)ââd Ď(f,x,x ) ^d whose i-th entry Ďi _i is the signed contribution of feature i. Most methods we discuss are additive, meaning they aim to satisfy the local accuracy (or completeness, or efficiency) property: âi=1dĎi=fâ(x)âfâ(xâ˛)=Îâfâ(x;xâ˛). _i=1^d _i\;=\;f(x)-f(x )\;=\; f(x;x ). (4) Equation (4) is the closest the field has to a universal target; we will return to it as Axiom 5 in Section I. Lundberg and Lee [62] formalized the class of additive feature attribution methods as those that admit an explanation surrogate gâ()=Ď0+âi=1dĎiâi,â0,1d,g( z)= _0+ _i=1^d _i\, z_i, zâ\0,1\^d, (5) where Ď0=fâ(xâ˛) _0=f(x ) is the offset. Setting = z= 1 recovers (4). This linear-in-mask surrogate is the common language of LIME, SHAP, DeepLIFT, LRP, and Integrated Gradients; their differences lie in how the coefficients Ďi _i are computed, not in the form of g. I-F A Running Example: Disagreement Without Error A two-feature interaction already shows why attribution methods can disagree without either method being erroneous. Let fâ(x1,x2)=x1âx2,x=(1,1).f(x_1,x_2)=x_1x_2, x=(1,1). With the zero baseline x0â˛=(0,0)x _0=(0,0), straight-line IG follows Îłâ(Îą)=(Îą,Îą)Îł(Îą)=(Îą,Îą) and gives IG1âĄ(x;x0â˛)=IG2âĄ(x;x0â˛)=âŤ01ÎąâÎą=12.IG_1(x;x _0)=IG_2(x;x _0)= _0^1Îą\,dÎą= 12. The corresponding single-reference Shapley value with vx0â˛refv^ref_x _0 also assigns (1/2,1/2)(1/2,1/2), because neither feature has value without the other. Now change only the reference point to x1â˛=(1,0)x _1=(1,0). The same model and same prediction give IGâĄ(x;x1â˛)=(0,1),IG(x;x _1)=(0,1), because the path changes only the second coordinate and decomposes fâ(1,1)âfâ(1,0)=1f(1,1)-f(1,0)=1. Both answers are valid relative to their baselines; they answer different comparison questions. If, in addition, the data distribution has X2=X1X_2=X_1, then the coalition inputs (1,0)(1,0) and (0,1)(0,1) used by interventional Shapley or occlusion are off the data manifold. A conditional or manifold-aware value function therefore changes the interpretation again, not because the model changed but because the meaning of âfeature absenceâ changed. TABLE IV: Running example for fâ(x1,x2)=x1âx2f(x_1,x_2)=x_1x_2 at x=(1,1)x=(1,1). Different specifications answer different questions. Specification Attribution Interpretation IG, xâ˛=(0,0)x =(0,0) (1/2,1/2)(1/2,1/2) interaction split evenly IG, xâ˛=(1,0)x =(1,0) (0,1)(0,1) credit for changing x2x_2 only Single-reference Shapley, xâ˛=(0,0)x =(0,0) (1/2,1/2)(1/2,1/2) coalition interaction split evenly Off-manifold perturbation under X2=X1X_2=X_1 specification-dependent masked inputs leave suppâ(pX)supp(p_X) I-G Notation Summary Table V summarizes the notation used throughout the rest of the paper. TABLE V: Notation used throughout the survey. Symbol Meaning f Predictive model ââX x,xix,x_i Input vector and i-th feature d Number of features N Feature index set 1,âŚ,d\1,âŚ,d\ xâ˛x Baseline / reference input xâ˛D_x Reference distribution Îâfâ(x;xâ˛) f(x;x ) fâ(x)âfâ(xâ˛)f(x)-f(x ) SâNS N Coalition of present features vâ(S)v(S) Value function on coalitions Îłâ(Îą)Îł(Îą) Path from baseline to input z Perturbation mask in [0,1]d[0,1]^d Ď=(Ďi) Ď=( _i) Attribution vector in âdR^d g Additive explanation surrogate pXp_X Data distribution âfâ f Gradient of model w.r.t. input I Axiomatic Foundations This section catalogues the axioms used to characterize feature attribution methods. The axioms originate in two distinct traditions: the cooperative-game tradition derived from Shapleyâs 1953 paper [82], and the path-method tradition introduced by Sundararajan, Taly, and Yan in their 2017 paper on Integrated Gradients [93]. A central mathematical observation of this survey, made explicit in Section VIII, is that these two axiom systems are not independent: many path-method axioms have direct game-theoretic analogues, and some methods (notably DeepLIFT and SHAP variants) can be characterized in either language. I-A Shapley Axioms Let v:2Nââv 2^N be a value function with vâ(â )=0v( )=0. A value ĎĎ assigns to each player iâNiâ N a real number Ďiâ(v) _i(v). Axiom 1 (Efficiency / Completeness). âi=1dĎiâ(v)=vâ(N) _i=1^d _i(v)=v(N). Axiom 2 (Symmetry). If players i,ji,j are interchangeable in v, meaning vâ(SâŞi)=vâ(SâŞj)v(SâŞ\i\)=v(SâŞ\j\) for every SâNâi,jS N \i,j\, then Ďiâ(v)=Ďjâ(v) _i(v)= _j(v). Axiom 3 (Dummy / Null Player). If vâ(SâŞi)=vâ(S)v(SâŞ\i\)=v(S) for every SâNâiS N \i\, then Ďiâ(v)=0 _i(v)=0. Axiom 4 (Additivity / Linearity). For any two value functions v1,v2v_1,v_2 and any Îą,βââÎą,β , Ďiâ(Îąâv1+βâv2)=ÎąâĎiâ(v1)+βâĎiâ(v2) _i(Îą\,v_1+β\,v_2)=Îą\, _i(v_1)+β\, _i(v_2). Result 1 (Shapley, 1953 [82]). Assumptions: a finite player set N=1,âŚ,dN=\1,âŚ,d\; a real-valued set function v:2Nââv 2^N with vâ(â )=0v( )=0. Under these assumptions, there is a unique value ĎĎ on N satisfying Axioms 1â4 for all such v, given by Ďiâ(v)=âSâNâi|S|!â(dâ|S|â1)!d!â[vâ(SâŞi)âvâ(S)]. _i(v)= _S N \i\ |S|!\,(d-|S|-1)!d! [v(SâŞ\i\)-v(S) ]. (6) The quantity in brackets, vâ(SâŞi)âvâ(S)v(SâŞ\i\)-v(S), is the marginal contribution of feature i to coalition S. The weight |S|!â(dâ|S|â1)!/d!|S|!\,(d-|S|-1)!/d! is the probability that, under a uniformly random ordering of the d players, exactly the members of S precede i; the Shapley value is thus the expected marginal contribution of i over random arrival orders. The uniqueness half of Result 1 follows from a basis argument that is worth recording because it recurs in later characterizations. The unanimity games uTâ(S)=â[TâS]u_T(S)=1[T S], for nonempty TâNT N, form a basis of the (2dâ1)(2^d-1)-dimensional space of value functions with vâ(â )=0v( )=0. On a unanimity game, the dummy axiom forces Ďiâ(uT)=0 _i(u_T)=0 for iâTiâ T, and symmetry with efficiency forces Ďiâ(uT)=1/|T| _i(u_T)=1/|T| for iâTiâ T. Linearity then determines ĎĎ on every v=âTcTâuTv= _Tc_Tu_T, and evaluating the resulting expression coalition by coalition yields (6). The full argument is in the original paper [82]. I-B Additive Feature Attribution Recall the additive surrogate of (5). Lundberg and Lee [62] showed that a single triple of axioms forces any method expressible as a linear-in-mask surrogate to coincide with Shapley values. Axiom 5 (Local Accuracy). gâ()=fâ(x)g( 1)=f(x), i.e. âiĎi+Ď0=fâ(x) _i _i+ _0=f(x). Axiom 6 (Missingness). For any i with xi=xiâ˛x_i=x _i, Ďi=0 _i=0. Axiom 7 (Consistency [62, 61]). If fâ˛f is a model such that the marginal contribution of feature i in fâ˛f is at least its marginal contribution in f for every coalition, then Ďiâ(fâ˛,x)âĽĎiâ(f,x) _i(f ,x)⼠_i(f,x). Result 2 (Lundberg and Lee, 2017 [62], Thm. 1). Assumptions: the attribution takes the additive surrogate form (5) for a binary mask â0,1d zâ\0,1\^d; a value function vf,xv_f,x over coalitions of features is fixed. Under these assumptions, the unique solution satisfying Axioms 5â7 coincides with the Shapley value Ďi=ĎiShapâ(vf,x) _i= _i^Shap(v_f,x). The proof is Theorem 1 of [62]. This is the formal sense in which SHAP unifies the additive attribution family. The unification, however, leaves open the choice of vf,xv_f,x, and as we saw in Section I that choice is itself consequential. I-C Alternative Cooperative-Game Values The finite-player Shapley value is the dominant attribution index in XAI, but it is not the only cooperative-game value relevant to explanation. The Banzhaf value [12] averages marginal contributions under a different coalition weighting scheme, placing equal weight on coalitions, while the Shapley value averages player arrival orders. Owenâs multilinear extension [70] connects finite games to polynomial extensions on the unit cube and is one route by which sampling and interaction calculations can be studied analytically. For continuous populations and cost-sharing problems, the Aumann-Shapley value [8] replaces finite coalitions with pathwise marginal rates. Specialized to a differentiable cost function f on âdR^d with reference point xâ˛x , the Aumann-Shapley charge to coordinate i is the diagonal-path integral ĎiASâ(x;xâ˛)=(xiâxiâ˛)ââŤ01âfâxiâ(xâ˛+Îąâ(xâxâ˛))âÎą,Ď^AS_i(x;x )=(x_i-x _i) _0^1 â fâ x_i (x +Îą(x-x ) )\,dÎą, (7) which is exactly the Integrated Gradients operator of Section V. This connection is substantive: it identifies IG as a fixed-path, feature-level cost-sharing rule, with the baseline and the straight-line path replacing the cooperative gameâs population model. I-D Path-Method Axioms The axiom set of Sundararajan et al. [93] targets attribution methods defined for differentiable models. Axiom 8 (Sensitivity-(a)). If the input x and baseline xâ˛x differ in exactly one feature and the predictions differ (fâ(x)â fâ(xâ˛)f(x)â f(x )), then that feature must receive a nonzero attribution. Axiom 9 (Sensitivity-(b)). If f does not mathematically depend on feature i, then Ďi=0 _i=0. (This is the differentiable analogue of dummy.) Axiom 10 (Implementation Invariance). If two networks f1f_1 and f2f_2 compute the same function (f1â(x)=f2â(x)f_1(x)=f_2(x) for all x), their attributions are identical. Axiom 11 (Completeness). âiĎi=fâ(x)âfâ(xâ˛) _i _i=f(x)-f(x ). Axiom 12 (Linearity). ĎĎ is linear in f. Axiom 13 (Symmetry-Preserving). If x and xâ˛x are symmetric in two features i,ji,j (xi=xjx_i=x_j and xiâ˛=xjâ˛x _i=x _j) and f is symmetric in those features, then Ďi=Ďj _i= _j. Result 3 (Sundararajan et al., 2017 [93]; further axiomatic characterizations by Lundstrom and Razaviyayn, 2025 [63]). Assumptions: f is differentiable along the straight-line path Îłâ(Îą)=xâ˛+Îąâ(xâxâ˛)Îł(Îą)=x +Îą(x-x ); the gradient is integrable on Îąâ[0,1]Îąâ[0,1]. Under these assumptions and within the attribution classes considered in the cited papers, Integrated Gradients along the straight-line path is singled out by Axioms 8â10 together with completeness, linearity, and symmetry-preservation. Lundstrom and Razaviyayn [63] give three additional independent axiomatic characterizations under varied axiom subsets. That several different axiom combinations identify the same straight-line operator strengthens the case for IG as a canonical member of the path family, but it does not make the operator assumption-free: each characterization fixes the attribution class and the path in advance. I-E Conservation and Backpropagation Axioms Layer-wise relevance propagation (LRP) [9] and DeepLIFT [83] are characterized not by a coalition or a path but by a conservation rule that distributes a quantity through the networkâs computation graph. Axiom 14 (Layer-wise Conservation [9]). For every layer â , the sum of relevances assigned to its inputs equals the sum of relevances received from its outputs: âiâinâ(â)Ri(â)=âjâoutâ(â)Rj(â+1). _i ( )R_i^( )= _j ( )R_j^( +1). (8) At the output layer, R(L)=fâ(x)R^(L)=f(x). At the input layer, âiRi(0)=fâ(x) _iR_i^(0)=f(x), so layer-wise conservation implies completeness. Axiom 15 (Summation-to-Delta [83]). DeepLIFT attributions CÎâxiâÎâyC_ x_i y satisfy âiCÎâxiâÎây=Îây _iC_ x_i y= y, where Îây=fâ(x)âfâ(xâ˛) y=f(x)-f(x ). Although LRP and DeepLIFT are sometimes presented as alternatives to Integrated Gradients, all three satisfy a completeness axiom of the form âiĎi=Îây _i _i= y. They differ in how the distribution is performed, not in what is preserved. I-F Robustness and Stability Axioms Beyond the classical axioms, several authors have proposed properties that capture an attribution methodâs robustness to small perturbations. Axiom 16 (Lipschitz Stability [5]). There exists L<âL<â such that âĎâ(x1)âĎâ(x2)ââ¤Lââx1âx2â\| Ď(x_1)- Ď(x_2)\|⤠L\,\|x_1-x_2\| for all x1,x2x_1,x_2. Axiom 17 (Continuity). Ď Ď is continuous in x (a strictly weaker condition than Axiom 16). These properties are not satisfied by raw gradient saliency in practice [39, 30], and their failure is one of the principal motivations for SmoothGrad and similar averaging methods (Section VI). Layer-wiseconservation(Ax. 14)Summation-to-delta(Ax. 15)CompletenessâiĎi=Îâf _i _i= fSensitivity-(a)(Ax. 8)one-feature differenceLipschitzstability(Ax. 16)Continuityin x(Ax. 17)Implementationinvariance(Ax. 10)Symmetry-preservationindep.impliesconditionalindependent Figure 2: Inter-axiom relationships. Layer-wise conservation (Axiom 14) and summation-to-delta (Axiom 15) each imply completeness. Completeness implies Sensitivity-(a) only in the Sensitivity-(a) setup, where x and xâ˛x differ in exactly one feature. Lipschitz stability is strictly stronger than continuity. Implementation invariance and symmetry-preservation are logically independent (Remark 1); the converses of all displayed implications fail in general. I-G Inter-axiom Relationships Many axioms imply or are implied by others. We collect the most useful relationships here for reference; they will recur in Section VIII. Remark 1 (Inter-axiom implications). The following implications follow from the definitions of the axioms; we list them as observations. (i) Completeness â Sensitivity-(a), under the Sensitivity-(a) setup: if x and xâ˛x differ in exactly one feature and fâ(x)â fâ(xâ˛)f(x)â f(x ), the sum constraint forces that changed feature to receive nonzero attribution. Without the one-feature-difference condition, completeness implies only that at least one feature has nonzero attribution. (i) Layer-wise Conservation â Completeness: summing the per-layer conservation from output to input yields âiRi(0)=fâ(x) _iR_i^(0)=f(x). (i) Lipschitz Stability â Continuity in x by definition. (iv) Implementation Invariance and symmetry-preservation are logically independent; neither implies the other in general. The converses generally fail: a method may be continuous without being Lipschitz, may satisfy completeness without being implementation invariant (e.g., a discrete-gradient method on a ReLU network), and may satisfy Sensitivity-(a) without satisfying completeness (e.g., raw saliency). The path forward in the next four sections is to take each major method family and identify, with reference to this axiom catalogue, exactly which axioms it does and does not satisfy. The resulting matrix, collected in Section VIII, is a central comparison table of this survey. IV Shapley and Cooperative-Game Attribution The Shapley value [82], originally developed to distribute the joint payoff of a cooperative game among its players, is the most extensively studied attribution method. Its appeal is axiomatic: it is the unique solution to a small set of intuitive constraints (Result 1). Its difficulty is computational: exact evaluation has cost exponential in the number of features. The methods surveyed in this section can be understood as different points on the trade-off curve between fidelity to the axiomatic Shapley value and the cost of approximating it. IV-A Exact Shapley Values For a value function v on 2N2^N, the Shapley value Ďi _i is given by (6). Equivalently, Ďiâ(v)=1d!ââĎâÎ â(N)[vâ(PreiĎâŞi)âvâ(PreiĎ)], _i(v)= 1d! _Ďâ (N) [v(Pre_i^ĎâŞ\i\)-v(Pre_i^Ď) ], (9) where Î â(N) (N) is the set of permutations of N and PreiĎPre_i^Ď is the set of features that precede i in Ď. The two forms agree by a counting argument: a fixed coalition SâNâiS N \i\ arises as PreiĎPre_i^Ď for exactly |S|!â(dâ|S|â1)!|S|!\,(d-|S|-1)! of the d!d! permutations (order the members of S, place i, order the rest), so grouping the sum in (9) by the value of PreiĎPre_i^Ď recovers the weights of (6). The form (9) is more convenient for sampling-based approximation: drawing permutations uniformly at random gives an unbiased Monte Carlo estimator whose variance decays at the standard Oâ(1/M)O(1/M) rate in the number of sampled permutations M. The principal cost of exact Shapley is the 2d2^d evaluations of v, plus the work of evaluating v itself, which under either the interventional or conditional definitions ((1)â(2)) requires in turn a model call or an expectation. For d>20d>20 exact evaluation is generally infeasible, and the methods below trade an approximation error for computational tractability. IV-B Pre-SHAP Game-Theoretic Feature Contributions Modern SHAP terminology can obscure an older line of work that already treated local explanation as cooperative-game credit allocation. Ĺ trumbelj and Kononenko [90] formulated individual classification explanations as feature-value contributions whose sum equals the change from an expected output to the modelâs prediction, and proposed a sampling approximation to avoid enumerating all feature subsets. This places black-box individual explanation in the Shapley lineage before the later SHAP unification. Quantitative Input Influence (QII) [25] developed a related but distinct transparency framework: it asks how much an input or group of inputs influences an output under specified interventions or perturbation distributions. QII is important in this survey because it separates the influence query from the estimator. That separation anticipates the value-function distinction that now dominates Shapley explanations: a numerical attribution is meaningful only after the intervention, conditioning, or perturbation semantics have been fixed. IV-C KernelSHAP KernelSHAP [62] reformulates Shapley value estimation as weighted linear regression. Sampling masks âźpShapâ() z p_Shap( z) with the Shapley kernel ĎShapâ()=dâ1(d||)â||â(dâ||), _Shap( z)= d-1 d| z|\,| z|(d-| z|), (10) KernelSHAP fits the additive surrogate gâ()g( z) of (5) by weighted least squares with weights ĎShap _Shap: Ď^=argâminĎ0,âŚ,Ďdâââ0,1dĎShapâ()â(vâ()âĎ0ââi=1dĎiâi)2, Ď= *arg\,min_ _0,âŚ, _d _ zâ\0,1\^d _Shap( z) (v( z)- _0- _i=1^d _i z_i )^2, (11) where vâ()v( z) is the model evaluated on the masked input under the chosen value function. The kernel weight (10) is infinite at ||â0,d| z|â\0,d\; those two masks are handled as the hard constraints Ď0=vâ() _0=v( 0) and âiĎi+Ď0=vâ() _i _i+ _0=v( 1), the latter being completeness. Lundberg and Lee proved that the solution of (11) over the full mask distribution coincides exactly with the Shapley values of (6). In practice, MâŞ2dM 2^d masks are sampled, yielding an estimator with bias and variance that have since been characterized explicitly by Covert and Lee [21] and refined by the unbiased estimator of [20]. KernelSHAP is model-agnostic: it requires only black-box access to f, making it applicable to gradient-free models. Its correctness, however, depends on the value function used in the regressionâs prediction targets. By default KernelSHAP marginalizes over a fixed reference (the interventional value function), and as a result it inherits all of the assumptions about feature independence discussed in Section I. IV-D TreeSHAP For tree ensembles, the structural recursion of the model can be exploited to compute Shapley values exactly in polynomial time. Lundberg et al. [61] introduced TreeSHAP, which runs in Oâ(TâLâD2)O(TLD^2) time on a tree ensemble with T trees, L leaves, and depth D. The algorithm maintains, at each node, a polynomial in âcoalition massâ that encodes the contribution of each subtree to all possible coalitions; this polynomial is propagated through the tree and combined at the leaves. TreeSHAP supports two value functions: path-dependent (the empirical conditional expectation along training-time tree splits) and interventional (the marginal value over a reference dataset). The two yield different attributions whenever features are correlated, a discrepancy now widely recognized in the practical SHAP literature [49, 92]. TreeSHAP also satisfies the consistency axiom (Axiom 7), a property that fails for naive gain-based feature importance. IV-E DeepSHAP and GradientSHAP DeepSHAP [62] extends DeepLIFT-Rescale rules to approximate Shapley values for deep networks. The key insight is that DeepLIFTâs per-neuron multipliers can be interpreted as expectations over a baseline distribution, and aggregated by linear combination to yield a Shapley-style attribution. The approximation is exact for linear models and for the composition of linear and ReLU layers with a single fixed baseline; for more complex architectures it is an empirical heuristic. GradientSHAP averages Integrated Gradients computations over a distribution of baselines drawn from the data [62, 33]. Specifically, ĎiGradSHAP _i^GradSHAP =xâ˛âźpX,ÎąâźUâ[0,1][(xiâxiâ˛) =E_x p_X,\,Îą U[0,1]\! [(x_i-x _i) (12) â âfâ(xâ˛+Îąâ(xâxâ˛))âxi]. ¡ â f(x +Îą(x-x ))â x_i ]. The expectation over xâ˛x is the link to Shapley reasoning, as discussed in detail by Erion et al. [33]; the expectation over Îą is the standard Integrated Gradients path integral. The combination yields a sampling-based estimator of the expected Shapley value with respect to the data distribution. IV-F SAGE (boundary case): Global Shapley Effects Scope note. SAGE produces a global feature-importance measure, one level above the local-attribution scope of this survey. We include it because it shares the Shapley axiomatic foundation with KernelSHAP and TreeSHAP and because its loss-based value function illustrates how the same axiomatic machinery extends beyond per-prediction attribution. We mark SAGE as a boundary case in the axiom matrix (Table IX) and treat its remaining global-explanation siblings as out of scope (Section XIII). The methods above attribute a single prediction. SAGE (Shapley Additive Global ExplanationS) [22] extends the framework to a global feature-importance measure by replacing the per-input value function with a loss-based one: vSAGEâ(S)=â(X,Y)â[ââ(fâ(XS,XSÂŻâ˛),Y)]+const.v^SAGE(S)=-E_(X,Y)\! [ (f(X_S,X_ S ),Y) ]+const. (13) Here XSÂŻâ˛X_ S marginalizes over the missing features. Shapley values of vSAGEv^SAGE measure how much of the modelâs predictive performance is attributable to each feature globally. SAGE inherits the same independence concerns as KernelSHAP but resolves them at the dataset level; KernelSHAP resolves them per prediction. It is tempting to treat any global Shapley score as a feature-selection criterion, but this is a separate modelling decision. Fryer, StrĂźmke, and Nguyen [38] show through counterexamples that the classical Shapley axioms do not by themselves guarantee suitability for subset selection; the game formulation must match the statistical objective of the selected feature set. IV-G Shapley-Taylor Interaction Indices Standard Shapley values attribute the prediction to single features. In many applications, including proteinâprotein interactions, drug combinations, and NLP feature interactions, joint effects are central. Dhamdhere et al. [28] introduced the Shapley-Taylor interaction index, generalizing Shapley to subsets: ĎTST,kâ(v)=âSâNâTw|S|,dkâÎTâvâ(S),Ď^ST,k_T(v)= _S N Tw^k_|S|,d\, _Tv(S), (14) where TâNT N with |T|â¤k|T|⤠k, ÎT _T is the |T||T|-th discrete derivative, and wkw^k is a generalization of the Shapley weight. The index is uniquely characterized by an axiom set that extends the classical Shapley axioms (Theorems 3 and 4 of [28]), and reduces to Shapley values when k=1k=1 and |T|=1|T|=1. Janizek, Sturmfels, and Lee [48] give a continuous analogue via second-order path integrals (Integrated Hessians), which we cover in Section V; the discrete and continuous interaction frames are related by Theorem 5 of [48]. Interaction discovery can also be approached directly from learned model structure. Tsang et al. [95] detect statistical interactions from neural-network weights, making the interaction object explicit without deriving it solely from local feature perturbations. At the model-element level, Neuron Shapley [40] applies Shapley valuation to neurons or filters as the players. Both works are boundary cases for this surveyâs local input-attribution scope, but they sharpen an important point: the âplayersâ in a cooperative explanation game need not be raw input coordinates. IV-H Interventional vs. Conditional Value Functions In Shapley-based attribution, the value function is often the most consequential modelling choice. The choice between vintv^int ((1)) and vcondv^cond ((2)) determines the answers to questions that look identical at the level of an explanation but differ at the level of inference. Interventional Shapley answers the question: âif I forced feature i to take its baseline value, how would the prediction change?â This is the natural quantity for debugging, mechanism discovery, and causal reasoning under the implicit assumption that the features can be intervened on independently. Conditional Shapley answers: âconditional on observing xix_i, how does the expected prediction change?â This is the natural quantity for predictive importance under the observed data distribution. The two coincide only when features are mutually independent. In the presence of correlated features, which is the usual case, they differ, and the gap can be large. Janzing, Minorics, and BlĂśbaum [49] argued that the conditional value function conflates association with causation and recommends the interventional value for almost all use cases. Aas, Jullum, and Løland [1] took the opposite position for risk-management applications, arguing that an interventional value function evaluates the model at points the data manifold never visits, producing implausibly large attributions for highly correlated features. Frye et al. [37] proposed manifold-aware Shapley values that compute the interventional quantity but only over on-manifold coalition completions. Sundararajan and Najmi [92] catalogued no fewer than four distinct âShapley values for model explanationâ: the conditional expectation, conditional expectation w.r.t. the model, the baseline expectation, and the random baseline expectation. They showed that they disagree on simple, low-dimensional examples. Their recommendation, which we endorse, is that any paper presenting Shapley attributions should explicitly name which of these is being computed. IV-I Estimation: Variance, Bias, and Recent Algorithms Sampling-based Shapley estimators introduce both bias and variance. Chen et al. [20] surveyed the estimator landscape, distinguishing 1. Permutation sampling, which gives an unbiased Monte Carlo estimator of (9); 2. KernelSHAP-style weighted regression, which is more sample-efficient on well-conditioned value functions; 3. Unbiased KernelSHAP [21], which corrects the bias introduced by paired sampling. For tree models, TreeSHAP is exact; for differentiable models, GradientSHAP is approximate but inexpensive; for arbitrary models, KernelSHAP is the default. The choice is again a value-function question: each estimator is optimal for a slightly different definition of the underlying Shapley value, as Figure 3 of Chen et al. makes explicit. IV-J Summary Table VI summarizes the Shapley-family methods discussed above. The table makes the central point of this section: the methods agree on what they aim to compute, namely a Shapley decomposition of the prediction, but differ in their value function, their approximation algorithm, and the assumptions under which they are exact. TABLE VI: Shapley-value variants for feature attribution. The âValue functionâ column refers to the definitions in (1)â(3). âExactâ means the method recovers the Shapley value without sampling, under the stated value function. Method Value function Estimator Complexity Feature dependence Exact Shapley any exhaustive Oâ(2d)O(2^d) handled by choice of v KernelSHAP interventional (default) weighted regression Oâ(Mâd)O(Md) per sample ignored Unbiased KernelSHAP [21] interventional paired sampling Oâ(Mâd)O(Md) ignored TreeSHAP [61] path-dep. or interventional exact recursion Oâ(TâLâD2)O(TLD^2) path-dependent option DeepSHAP [62] multiplier-based DeepLIFT propagation Oâ(net)O(net) approximation GradientSHAP [33] expected baseline path + baseline avg. Oâ(KâM)O(KM) via baseline distribution SAGE [22] loss-based, global permutation sampling Oâ(Mâd2)O(Md^2) via marginalization Shapley-Taylor [28] generalizes interventional enumeration up to order k Oâ((dk))O\! ( dk ) ignored at order k V Path-Based Attribution Path-based methods compute attributions by integrating the gradient of f along a continuous path in input space from a baseline xâ˛x to the target input x. They are the continuous analogue of cooperative-game methods: where Shapley values average marginal contributions over discrete coalitions, path methods average gradients over a continuous trajectory. The canonical instance, Integrated Gradients, was introduced by Sundararajan, Taly, and Yan [93] and remains the most widely used member of the family. V-A Integrated Gradients Let f be differentiable. Integrated Gradients (IG) attributes to feature i the quantity IGiâĄ(x;xâ˛)=(xiâxiâ˛)ââŤ01âfâ(xâ˛+Îąâ(xâxâ˛))âxiâÎą.IG_i(x;x )=(x_i-x _i) _0^1 â f (x +Îą(x-x ) )â x_i\,dÎą. (15) The integral is taken along the straight-line path Îłâ(Îą)=xâ˛+Îąâ(xâxâ˛)Îł(Îą)=x +Îą(x-x ), evaluated by Riemann sums in practice (typically Kâ[20,300]Kâ[20,300] steps). Equation (15) satisfies completeness exactly: âi=1dIGiâĄ(x;xâ˛)=fâ(x)âfâ(xâ˛). _i=1^dIG_i(x;x )=f(x)-f(x ). (16) Writing gâ(Îą)=fâ(Îłâ(Îą))g(Îą)=f(Îł(Îą)), the chain rule gives gâ˛â(Îą)=âiâifâ(Îłâ(Îą))â(xiâxiâ˛)g (Îą)= _i _if(Îł(Îą))\,(x_i-x _i) because Îłiâ˛â(Îą)=xiâxiⲠ_i (Îą)=x_i-x _i on the straight line, and so fâ(x)âfâ(xâ˛)=gâ(1)âgâ(0)=âŤ01gâ˛â(Îą)âÎą=âi=1dIGiâĄ(x;xâ˛),f(x)-f(x )=g(1)-g(0)= _0^1g (Îą)\,dÎą= _i=1^dIG_i(x;x ), (17) the middle equality being the fundamental theorem of calculus and the last exchanging the finite sum with the integral. Result 4 (Sundararajan et al. [93]). Assumptions: f is differentiable on the straight-line path between xâ˛x and x; the gradient âf/âxiâ f/â x_i is integrable on [0,1][0,1]. Under these assumptions, Integrated Gradients along the straight-line path satisfies sensitivity-(a), sensitivity-(b), implementation invariance, completeness, linearity, and symmetry-preservation (Axioms 8â10 plus completeness and symmetry). Failures of differentiability (e.g., ReLU kink points) are handled by the standard interpretation of partial derivatives as subgradients almost everywhere along the path. Implementation invariance follows from the fact that âf/âxiâ f/â x_i depends only on the input-output behaviour of f, not on the networkâs parametric form. Completeness follows from the chain rule. Sensitivity-(a) follows because the integrand is nonzero whenever fâ(x)â fâ(xâ˛)f(x)â f(x ) along Îł. These properties are robust under any monotone reparameterization of the path, but not under arbitrary changes to Îł: implementation invariance, in particular, fails for paths that depend on the modelâs parametrization, an observation made precise by Lundstrom and Razaviyayn [63]. V-B Baselines and Path Sensitivity The IG attribution depends on the baseline xâ˛x . This dependence is mathematically necessary to define Îâf f and is substantively important. The choice of xâ˛x encodes what counts as the âabsenceâ of a feature. Sturmfels, Lundberg, and Lee [91] catalogued the most common baselines and their failure modes: ⢠The zero baseline xâ˛=x = 0 is the default in many implementations but is often pathological. In image models, black pixels are themselves features, and Sensitivity-(a) is then satisfied in a misleading way: every dark pixel in x receives an attribution of magnitude |xiâ âf/âxi||x_i¡â f/â x_i| that conflates âimportant to the modelâ with ânot equal to zeroâ. ⢠The mean baseline xâ˛=â[X]x =E[X] has the merit of being on or near the data manifold but can still be highly atypical. For ImageNet, the mean is a uniform grey image. ⢠The expected baseline, due to Erion et al. [33], replaces a fixed xâ˛x with an expectation over the training distribution: EGiâ(x) _i(x) =xâ˛âźpX,ÎąâźUâ[0,1][(xiâxiâ˛) =E_x p_X,\,Îą U[0,1]\! [(x_i-x _i) (18) â âfâ(xâ˛+Îąâ(xâxâ˛))âxi]. ¡ â f(x +Îą(x-x ))â x_i ]. This was already encountered as (12); in the path-integral framing it is the natural extension of IG to a distribution of baselines. ⢠Adversarial or informative baselines are chosen so that fâ(xâ˛)f(x ) is itself meaningful (e.g., a baseline of the target class for a counterfactual attribution). x1x_1x2x_2suppâ(pX)supp(p_X)x0â˛x _0xblurâ˛x _blurxxstraight linefrom x0â˛x _0off-manifold segmenton-manifold pathfrom xblurâ˛x _blur Figure 3: Baseline and path dependence for path-based attribution in a two-feature input space. The shaded region is the support of the data distribution. The straight-line path from a zero baseline x0â˛x _0 spends its early segment (dashed) outside suppâ(pX)supp(p_X), where f extrapolates and gradients are unconstrained by training data. A path from an in-distribution baseline xblurâ˛x _blur that stays inside the support queries f only where it was trained. Both decompose the same total, âiĎi=fâ(x)âfâ(xâ˛) _i _i=f(x)-f(x ) for their respective xâ˛x , but the per-feature split depends on the baseline and the path. V-C Adaptive and Region-Based Paths The straight-line path is mathematically convenient but may be poorly matched to the data manifold. It traverses regions of X where the model may behave erratically, a well-known manifestation of gradient saturation in deep networks means âf/âxiâ0â f/â x_iâ 0 along long stretches of the path [83, 6], washing out attributions for important features. Guided Integrated Gradients (Guided IG) [52] replaces the straight-line path with an adaptive trajectory that steers around low-gradient regions. At each step, the path follows the direction of steepest absolute partial derivative, restricted to features that have not yet been âsaturatedâ. Guided IG retains completeness (it is still a path integral) but loses the symmetry-preservation property: a path that depends on the gradient landscape is no longer permutation-equivariant. XRAI [51] attributes credit to image regions, with individual pixels grouped by a segmentation procedure; each region is then assigned the sum of IG attributions of its constituent pixels, divided by region area to give an importance score. XRAI is best understood as an aggregation layer over IG; it does not change the underlying attribution but improves the human readability of image-domain explanations. Blur Integrated Gradients [100] replaces the path in input space with a path in scale space: Îłâ(Îą)Îł(Îą) is a sequence of progressively less blurred versions of x, with the heavily blurred input as the baseline. This avoids the pathologies of zero-baseline IG for images while keeping the path integral interpretable as a decomposition of Îâf f. The trade-off is that the resulting attribution is no longer additive in the original pixel coordinates; it lives in scale space. V-D Higher-Order Path Methods: Integrated Hessians To attribute joint effects of feature pairs, Janizek, Sturmfels, and Lee [48] introduced Integrated Hessians (IH). Where IG integrates âf/âxiâ f/â x_i along a single path from xâ˛x to x, IH integrates the second mixed partial â2f/âxiââxjâ^2f/â x_iâ x_j along a nested pair of paths and yields an interaction attribution Îiâj _ij satisfying âi,jÎiâjâ(x;xâ˛)=fâ(x)âfâ(xâ˛). _i,j _ij(x;x )=f(x)-f(x ). (19) Result 5 (Janizek et al. [48], Thm. 1). Assumptions: f is twice-differentiable along the nested path; the mixed partials â2f/âxiââxjâ^2f/â x_iâ x_j exist and are integrable. Under these assumptions, Integrated Hessians is the unique attribution of second-order feature interactions consistent with the IG axioms when applied to the gradient-of-IG operator, as stated in the cited paper. IH connects directly to the Shapley-Taylor interaction index [28] (Section IV): the discrete Shapley-Taylor index of order 2 is a coalition-sampling approximation of the IH continuous integral. V-E Conductance: Internal Path Attribution The path integral (15) attributes credit to input features. Conductance [29] extends the same construction to internal neurons: Condhâ(x;xâ˛)=(xâxâ˛)â¤ââŤ01âfâh|Îłâ(Îą)ââhâx|Îłâ(Îą)âdâÎą,Cond_h(x;x )=(x-x ) _0^1 â fâ h |_Îł(Îą) â hâ x |_Îł(Îą)dÎą, (20) where h is the activation of a chosen hidden neuron. By construction, conductances of all neurons in a layer sum to Îâf f, giving a layer-wise decomposition consistent with the LRP conservation axiom (Axiom 14) without imposing layer-specific rules. V-F Path Dependence: When Does the Path Matter? A central theoretical question is: how sensitive are IG and its variants to the choice of path? Every method in this section is an instance of the generalized path attribution IGiÎłâĄ(x;xâ˛)=âŤ01âfâxiâ(Îłâ(Îą))âÎłiâ˛â(Îą)âÎą,IG^Îł_i(x;x )= _0^1 â fâ x_i (Îł(Îą) )\, _i (Îą)\,dÎą, (21) for some differentiable Îł with Îłâ(0)=xâ˛Îł(0)=x and Îłâ(1)=xÎł(1)=x; the straight line recovers (15) and the Aumann-Shapley value (7). Summing (21) over i and applying the chain rule as in (17) shows that every path-integral method of this form satisfies completeness, provided f is differentiable along Îł with integrable gradient and the path has the stated endpoints: the total âŤ01âfâ(Îłâ(Îą))â¤âÎłâ˛â(Îą)âÎą=fâ(x)âfâ(xâ˛) _0^1â f(Îł(Îą)) Îł (Îą)\,dÎą=f(x)-f(x ) depends only on the endpoints. The per-feature terms of (21), however, are not endpoint-determined: the decomposition that IG, EG, Guided IG, and Blur IG return is path-dependent. Two paths between the same xâ˛x and x that disagree on the per-feature decomposition correspond to two different ways of attributing the same total credit. This is not necessarily an error; it reflects the fact that attribution is underdetermined without additional assumptions. Under the assumptions and attribution class used by Sundararajan et al. [93], the straight-line path is singled out by symmetry-preservation and related axioms, but other choices are defensible under different axiom sets [63]. V-G Summary Table VII summarizes the path-based methods. The common thread is that each method makes one of two changes to IG (15): either it changes the baseline (EG, GradientSHAP), or it changes the path (Guided IG, Blur IG, XRAI), or both (IH adds an additional integration dimension). The choice of path or baseline determines which axioms hold, which features are highlighted, and how robust the attribution is to gradient saturation. TABLE VII: Path-based attribution methods. All preserve completeness; they differ in the choice of path and baseline, with corresponding effects on the remaining axioms. Method Path Baseline Completeness Interactions Suited to Integrated Gradients [93] straight line fixed xâ˛x yes 1st order tabular, image Expected Gradients [33] straight line xâ˛âźpXE_x p_X yes 1st order tabular, image Guided IG [52] adaptive fixed xâ˛x yes 1st order image (noisy IG) Blur IG [100] scale space blurred input yes 1st order image XRAI [51] inherits IG path inherits IG baseline yes (per region) 1st order image (segmented) Integrated Hessians [48] nested paths fixed xâ˛x yes (sum over pairs) 2nd order feature interactions Conductance [29] straight line in x fixed xâ˛x yes (sum over neurons) 1st order internal neurons VI Gradient and Backpropagation Attribution Gradient and backpropagation methods derive an attribution from a single backward pass through the model. They are the cheapest family of attribution methods, typically one or two backward passes, and historically were the first to be applied to deep networks. They are also the family with the most diverse axiomatic profiles: some satisfy completeness, some do not; some are implementation-invariant, some are not; some are continuous in x, and a notable subset is so discontinuous as to be operationally unstable [39, 30]. VI-A Raw Saliency and Gradient Ă Input The earliest gradient-based attribution is the saliency map [84, 10]: Ďisalâ(x)=|âfâ(x)âxi|orĎisalâ(x)=âfâ(x)âxi. _i^sal(x)= | â f(x)â x_i | _i^sal(x)= â f(x)â x_i. (22) Saliency is the first-order Taylor coefficient of f at x. It satisfies sensitivity-(b) and implementation invariance but not completeness or sensitivity-(a): a feature can change the prediction substantially yet receive a gradient near zero whenever f is locally flat (gradient saturation [83]). The gradient Ă input variant, Ďigxiâ(x)=xiâ âfâ(x)âxi, _i^gxi(x)=x_i¡ â f(x)â x_i, (23) is the first-order term of the Taylor expansion of f around the origin. It is one of the cheapest sensible attributions, and is the K=1K=1 approximation of Integrated Gradients with xâ˛=x = 0. It satisfies completeness only when f is linear. VI-B SmoothGrad and Noise Averaging SmoothGrad [87] smooths the saliency map by averaging over noisy versions of the input: ĎiSGâ(x)=Ďľâźâ(0,Ď2âI)â[|âfâ(x+Ďľ)âxi|]. _i^SG(x)=E_Îľ (0,Ď^2I)\! [ | â f(x+Îľ)â x_i | ]. (24) The averaging effectively replaces f with the Gaussian-smoothed model f~Ď=fââ(0,Ď2âI) f_Ď=f*N(0,Ď^2I), i.e. f~Ďâ(x)=Ďľâ[fâ(x+Ďľ)] f_Ď(x)=E_Îľ[f(x+Îľ)]. For f locally integrable with gradient of at most polynomial growth, differentiation and expectation interchange, so âf~Ďâ(x)=Ďľâźâ(0,Ď2âI)â[âfâ(x+Ďľ)]:â f_Ď(x)=E_Îľ (0,Ď^2I) [â f(x+Îľ) ]: (25) the signed SmoothGrad average is an exact gradient of a smoothed model, and inherits whatever axioms hold for gradients of f~Ď f_Ď. The absolute-value form in (24) breaks this identity because [|â |]â |[â ]|E[|¡|]â |E[¡]|, so absolute SmoothGrad is not the gradient of any smoothed model. SmoothGrad substantially reduces the high-frequency artefacts visible in raw saliency maps for image classifiers, at the cost of a hyperparameter Ď that controls the trade-off between locality and stability. VarGrad [4] computes the variance rather than the expectation: ĎiVGâ(x)=VarĎľâ[âfâ(x+Ďľ)/âxi] _i^VG(x)=Var_Îľ[â f(x+Îľ)/â x_i]. VarGrad highlights features for which the gradient is locally unstable, often providing complementary information to SmoothGrad. Both methods can be viewed as Monte Carlo estimators of the gradient of a noise-smoothed model, and as such inherit the axiomatic profile of that smoothed model. They satisfy implementation invariance but typically not completeness, since the smoothed gradient does not integrate to fâ(x)f(x) in general. VI-C Guided Backpropagation and Deconvolution Guided backpropagation [88] and deconvolution networks [102] modify the backward pass through ReLU units. Standard backpropagation uses positive forward activations; these methods additionally suppress gradients of opposite sign. Result 6 (Empirical finding of Adebayo et al. [4]). On standard image classifiers, guided backpropagation produces attribution maps that are visually largely independent of the modelâs parameters: replacing the weights with random values yields visually similar attributions. We report this as an empirical result from the cited paper, not as a formal theorem. Result 6 (the model-randomization sanity check) is one of the strongest empirical critiques of guided backpropagation: an attribution that is largely insensitive to model parameters cannot, by itself, support strong claims about model-specific reasoning. We return to the sanity-check framework in Section IX. VI-D DeepLIFT DeepLIFT [83] replaces gradients with multipliers computed against a reference activation. For each neuron h with reference value hrefh_ref and difference Îâh=hâhref h=h-h_ref, the DeepLIFT contribution CÎâxâÎâhC_ x h propagates through the network analogously to the chain rule: CÎâxiâÎây=âhmxiâhâ CÎâhâÎây,C_ x_i y= _hm_x_ih¡ C_ h y, (26) where mxiâh=CÎâxiâÎâh/Îâxim_x_ih=C_ x_i h/ x_i is a multiplier. DeepLIFT satisfies the summation-to-delta axiom (Axiom 15) by construction. Two distinct rules, Rescale (for monotone nonlinearities) and RevealCancel (for pairs of opposing effects), determine the multiplier at each layer. Result 7 (Ancona et al. [6], §3). Assumptions: the network is composed only of linear layers and monotone elementwise nonlinearities (e.g., ReLU, sigmoid, tanh); there are no skip connections, attention layers, concatenations, or non-monotone operators; a single fixed baseline xâ˛x is used. Under these assumptions, DeepLIFT-Rescale and Integrated Gradients coincide in the small-input-increment limit. When the assumptions fail (Add, GroupNorm, attention, concatenation), the equivalence breaks and the methods can yield different attributions. Result 7 establishes a key bridge: under restricted architectures and propagation rules, DeepLIFT-Rescale and Integrated Gradients can be viewed as closely related approximations of the same path-based quantity. Outside those assumptions, they should be treated as distinct attribution operators. They differ in computational profile (one forward+backward pass for DeepLIFT vs. K for IG) and in their handling of non-monotone nonlinearities (the RevealCancel rule addresses a case IG does not). VI-E Layer-Wise Relevance Propagation LRP [9] propagates a quantity called relevance backward through the network, subject to the conservation axiom (Axiom 14). The relevance of an output neuron is initialized to fâ(x)f(x); at each layer, it is decomposed into contributions from the preceding layer using a propagation rule. The most common rules are: z-rule Ri=âjziâjâiâ˛ziâ˛âjâRjR_i= _j z_ij _i z_i jR_j, where ziâj=xiâwiâjz_ij=x_iw_ij is the pre-activation contribution. Numerically unstable near zero. Ͼξ-rule adds a small constant Ͼξ to the denominator to suppress numerically unstable terms. Îł-rule amplifies positive contributions with multiplier Îł, used for the upper layers of deep classifiers. The LRP composition strategy of Montavon et al. [66] applies different rules to different layers, producing attribution maps that are empirically more faithful to the modelâs behaviour than any single rule alone. LRP is implementation-invariant only for the standard rules applied to specific architecture families; bespoke rules can break the property. It satisfies conservation by construction, hence completeness. It does not in general satisfy Sensitivity-(b): an input feature that is formally a dummy can still receive nonzero relevance under certain rules. VI-F FullGrad FullGrad [89] extends gradient-based attribution to incorporate bias terms, which earlier methods ignored. The key observation is that for a network with biases, the gradient âxf _xf alone does not satisfy completeness; the bias contributions must be added explicitly. FullGrad computes ĎiFGâ(x)=xiââfâxi+ââfâbiasâ(x),Ď^FG_i(x)=x_i â fâ x_i+ _ f_ ^bias(x), (27) where fâbiasf_ ^bias is the contribution from biases at layer â . FullGrad satisfies completeness exactly: âiĎiFG=fâ(x) _i _i^FG=f(x). This target includes the bias contribution at xâ˛=x = 0 and therefore differs from Îâf f. VI-G The CAM Family: From Class Activation to Shapley-CAM Class Activation Mapping (CAM) [104] was originally developed for global-average-pooled CNNs as a coarse spatial map of which regions activate a given class. Grad-CAM [80] generalized CAM to arbitrary CNN architectures by using gradients of the class score with respect to the final convolutional feature map: Lc,(i,j)GC=ReLUâ(âkÎąkcâA(i,j)k),Îąkc=1Zââi,jâfcâA(i,j)k.L^GC_c,(i,j)=ReLU\! ( _k _k^c\,A^k_(i,j) ), _k^c= 1Z _i,j â f_câ A^k_(i,j). (28) Here AkA^k is the k-th feature map of the target convolutional layer and Z is the number of spatial positions. The weights Îąkc _k^c average the class-score gradient over space, and the final map is a ReLU-rectified sum of the feature maps. Grad-CAM++ [18] replaces the average pooling with a weighted average that emphasizes positive contributions: Îąkc,++ _k^c,++ =âi,jgkâiâj(2)2âgkâiâj(2)+âa,bA(a,b)kâgkâiâj(3), = _i,j g^(2)_kij2g^(2)_kij+ _a,bA^k_(a,b)g^(3)_kij, (29) gkâiâj(m) g^(m)_kij =âmfcâ(A(i,j)k)m. = â^mf_câ(A^k_(i,j))^m. The motivation is to handle multiple instances of the target class. Score-CAM [97] eliminates gradients entirely by computing weights through perturbation: each feature map is upsampled into a soft mask, applied to the input, and the resulting class score determines the weight. This avoids gradient saturation but adds Oâ(K)O(K) forward passes per attribution, where K is the number of channels. Ablation-CAM [26] computes weights by explicitly zeroing one channel at a time: Îąkc=(fcâfcâk)/fc _k^c=(f_c-f_c k)/f_c, where fcâkf_c k is the prediction with channel k ablated. LayerCAM [50] aggregates Grad-CAM-style maps across multiple layers, including layers before the final convolution, giving finer spatial resolution. Eigen-CAM [67] uses the principal components of the feature map matrix as weights, removing the class-conditional gradient entirely. HiResCAM [32] corrects a known failure of Grad-CAM: the ReLU-and-average step in (28) can produce attribution maps that do not faithfully reflect the modelâs prediction. HiResCAM eliminates the spatial pooling and applies the gradient directly, restoring faithfulness at the cost of some spatial smoothness. Shap-CAM [103] reframes the CAM weighting as a Shapley value over channels, providing an axiomatic justification for Score-CAMâs perturbation-based weighting. What unites the CAM family is a spatial projection of attribution onto the last convolutional layer. In their usual form, CAM variants are not pixel-level complete decompositions of fâ(x)âfâ(xâ˛)f(x)-f(x ); they are intermediate-layer attributions that are then upsampled to image resolution. VI-H Transformer Attribution: Attention Is Not Enough For transformer architectures, the most-cited attribution candidate is the attention map itself. The mathematical critique of [47, 81] showed that attention weights are not generally consistent with model behaviour: alternative attention distributions can produce identical or near-identical predictions, so attention is not uniquely identified by the modelâs output. Wiegreffe and Pinter [99] treat attention as one plausible explanatory hypothesis without claiming that it is unique. They propose testing whether the attention pattern is plausible under the model. Two attribution methods have since become standard for transformers: Attention rollout [2] aggregates attention weights across layers by treating each layerâs attention matrix as a transition probability and computing the rollout A~L=ââ=1L(A(â)+I)/2 A^L= _ =1^L(A^( )+I)/2. The product captures the iterated effect of attention across the depth of the model. Transformer relevance propagation [19] extends LRP to transformers, distributing the class score backward through both attention and feed-forward components. It satisfies a conservation axiom at each layer, restoring the property that attention alone does not. VI-I Summary Gradient and backpropagation methods range from raw saliency (one forward+backward pass, no axioms beyond sensitivity-(b)) to LRP and DeepLIFT (multiple rules, conservation, implementation invariance under the right conditions). They are united by their differentiability assumption and their computational efficiency, but divided on which axioms they satisfy and on how they behave under known sanity checks. Table VIII summarizes the family. TABLE VIII: Gradient and backpropagation attribution methods. âCostâ is in backward passes; âSanityâ is the model-randomization sanity check of [4]: pass means the attribution changes substantially when model weights are randomized. Method Derivative Completeness Impl. Inv. Sanity Noise smoothing Cost Saliency [84] 1st no yes pass no 1 Gradient Ă Input 1st only if linear yes pass no 1 SmoothGrad [87] 1st (smoothed) no yes pass yes N VarGrad [4] 1st (variance) no yes pass yes (variance) N Guided BP [88] 1st (rule-modified) no no fail no 1 Deconvnet [102] 1st (rule-modified) no no fail no 1 DeepLIFT [83] multipliers yes yes (Rescale) pass no 1 LRP [9] rule-based yes (conservation) yes (standard rules) pass no 1 FullGrad [89] 1st + bias yes (vs. 0 baseline) yes pass no 1 Grad-CAM [80] 1st of last conv no (region only) layer-dep. partial no 1 Grad-CAM++ [18] 1stâ3rd of last conv no layer-dep. partial no 1 Score-CAM [97] none (forward only) no layer-dep. pass no K fwd HiResCAM [32] 1st (no pooling) yes (per layer) layer-dep. pass no 1 VII Perturbation and Occlusion Attribution Perturbation methods sidestep the question of value functions, paths, and gradients by directly measuring how the prediction changes when parts of the input are modified. They are conceptually simple (âocclude a feature and see what happensâ) and fully model-agnostic, requiring only black-box access to f. Their mathematical content lies in how perturbations are chosen, how predictions on perturbed inputs are aggregated, and how their attributions should be interpreted given the off-manifold inputs they typically produce. VII-A Occlusion Sensitivity and Prediction Difference The simplest perturbation method is occlusion, introduced for CNNs by Zeiler and Fergus [102]: slide a fixed patch across the input image, record the change in the prediction at each position, and use the resulting map as an attribution. Formally, let pâ(x)P_p(x) denote x with a patch of features pâNp N replaced by a reference value: Ďpoccâ(x)=fâ(x)âfâ(pâ(x)).Ď^occ_p(x)=f(x)-f(P_p(x)). (30) Occlusion is a single-coalition Shapley estimate: it computes the marginal contribution of the patch p to the full feature set N, ignoring all other coalitions. It is unbiased when the features outside p are independent of those inside, and biased otherwise. The Prediction Difference Analysis of Zintgraf et al. [105] replaces the deterministic patch with a conditional expectation: features are âremovedâ by marginalizing over their conditional distribution given the rest of the input. This addresses the off-manifold issue at the cost of requiring a conditional density model. VII-B LIME: Local Linear Surrogates LIME [73] fits a local linear surrogate to f in a neighbourhood of x. Given a similarity kernel Ďxâ() _x( z) and a perturbation distribution over masks, LIME solves gâ=argâĄmingâââĎxâ()â(fâ(â(x))âgâ())2+Ίâ(g),g^*= _g _ z _x( z) (f(P_ z(x))-g( z) )^2+ (g), (31) where G is a family of interpretable models (typically sparse linear) and Ί is a complexity penalty. The coefficients of gâg^* are the attributions. LIME is a general additive surrogate method: any choice of mask distribution and kernel yields a method in the class (5). KernelSHAP is the special case in which the kernel is ĎShap _Shap of (10) and the regression is unregularized; in that case the surrogate coefficients coincide with the Shapley values [62]. Different kernels yield different attributions, and the LIME kernel (exponential of inverse cosine distance, in the original paper) does not satisfy the Shapley axioms. MAPLE [72] is a supervised-neighbourhood variant of the same local-surrogate idea. Instead of sampling an unsupervised neighbourhood around x, MAPLE uses tree-ensemble structure to weight training points and then fits a local linear model under that induced neighbourhood. This gives MAPLE a clearer statistical object than a generic proximity kernel: the local explanation is tied to the predictive neighbourhood learned by random forests [16] or boosted trees [36]. It also illustrates a general lesson for local surrogates: locality is not a purely geometric choice, but a modelling assumption about which perturbations should stand in for nearby counterfactuals. VII-C Anchors: Rule-Based Local Explanations Scope note. Anchors lie at the boundary of the local-additive scope of this survey: their output is a rule predicate, not a per-feature attribution vector, so the linear-in-mask surrogate of (5) does not apply directly. We include Anchors in this section because the underlying perturbation distribution is closely related to that of LIME and because Anchors is widely used as a baseline against attribution methods; we mark it as a boundary case in the axiom matrix (Table IX). Anchors [74] step away from the additive surrogate form entirely and produce a set of rules A such that, conditional on the rules holding, fâ(xâ˛)=fâ(x)f(x )=f(x) with high probability: ââ(fâ(xâ˛)=fâ(x)|Aâ(xâ˛)=1)âĽ1âδ,P (f(x )=f(x)\, |\,A(x )=1 )⼠1-δ, (32) for some tolerance δ and a coverage condition on the rule. The rules are themselves the explanation; they have no real-valued attribution per feature. Anchors and LIME together span the local-surrogate design space: a local linear model versus a local rule. VII-D Meaningful Perturbations and Extremal Masks Fong and Vedaldi [34] introduced meaningful perturbations, which optimize a smooth mask â[0,1]d zâ[0,1]^d that maximally suppresses the prediction subject to a sparsity constraint: minâĄfâ(â(x))+Îťââ1+ÎźâTVâ(), _ z\;f(P_ z(x))+Îť\| z\|_1+Îź\,TV( z), (33) where TVTV is a total-variation regularizer. The optimal mask identifies the regions of x whose removal most disrupts the prediction, yielding a localized form of attribution. The follow-up Extremal Perturbations [35] replaces the unconstrained â1 _1 objective with a hard area constraint: âfind the smallest mask of area a that maximizes/minimizes the predictionâ. This avoids sensitivity to the Lagrange multiplier Îť and yields more interpretable masks. VII-E RISE and Random-Mask Methods RISE [71] averages over a large number of random masks: ĎiRISEâ(x)=1Nâân=1Nfâ((n)â(x))âi(n)/p,Ď^RISE_i(x)= 1N _n=1^Nf(P_ z^(n)(x))\, z^(n)_i\,/\,p, (34) where p is the probability that feature i is unmasked. RISE is embarrassingly parallel and requires no gradient access, making it appealing for true black-box settings. It is a Monte Carlo estimator of the marginal expectation of f with respect to the mask distribution, normalized to give a per-feature score. VII-F Counterfactual Generation: FIDO-CA (boundary case) Scope note. FIDO-CA produces counterfactual generations and does not yield per-feature attribution scores; we include it here because the underlying optimisation formulation (mask + generative completion) lies on the same axis as the perturbation-distribution choice made by LIME and RISE, and because its on-manifold mask is a constructive answer to the off-manifold failure mode of mask-based attribution. Strictly speaking, the contribution sits in the counterfactual-explanation literature catalogued in Section XIII. Chang et al. [17] introduced FIDO-CA, which generates counterfactual perturbations using a generative model. Where RISE and meaningful perturbations replace masked features with constants or blurred values, FIDO-CA fills them with samples from a conditional generator Gâ(,x)G( z,x): ĎiFIDOâ(x)=argâĄminâlogâĄfcâ(Gâ(,x))+Îťââ1.Ď^FIDO_i(x)= _ z\;- f_c (G( z,x) )+Îť\| z\|_1. (35) The motivation is that the resulting perturbed inputs are on-manifold, addressing one of the principal failure modes of mask-based methods (Section X). VII-G Real-Time Saliency: Learned Mask Predictors Dabkowski and Gal [24] trained a separate network gĎg_Ď to predict the optimal mask of (33): gĎâ(x)âargâĄmaxâĄfcâ(â(x))âÎťââ1.g_Ď(x)â _ z\;f_c(P_ z(x))-Îť\| z\|_1. (36) The learned mask predictor gives near-instant attributions at inference time but introduces an additional model whose own validity must be verified. VII-H Information-Bottleneck Attribution Schulz et al. [79] cast mask-based attribution as restricting the information flow through the network. They add Gaussian noise to intermediate activations and optimize the noise variance per spatial location to maximize the suppression of information about the input while preserving the prediction. The resulting per-location noise scale is the attribution. The framing exposes a duality between mask-based attribution and information bottlenecks that is independently interesting; the cost is one optimization per attribution. VII-I Insertion / Deletion Evaluation Petsiuk et al. [71] also introduced the insertion / deletion curves that have since become a standard evaluation tool (Section IX). Features are added or removed in order of attribution magnitude; the area under the resulting prediction-versus-step curve quantifies how well the attribution identifies features that matter to the model. These curves are not an attribution method per se but a meta-evaluation built on any underlying attribution. VII-J Summary Perturbation methods are the natural complement to gradient methods. They make no differentiability assumption and require no backpropagation, but pay for that generality in two ways: many forward passes per attribution, and a strong dependence on the perturbation distribution. The choice of how to âremoveâ a feature (by zeroing, blurring, sampling from pXp_X, sampling from pâ(XâŁxSÂŻ)p(X x_ S), or generating with G) determines whether the resulting attribution is interventional, conditional, or on-manifold, and is a direct counterpart of the value-function choice in Shapley methods (Section IV). The same axes therefore organize both families. VIII Mathematical Comparison Framework Local additive feature attribution Ďâ(f,x,xâ˛)ââd Ď(f,x,x ) ^d Shapley / cooperative game Exact Shapley KernelSHAP TreeSHAP DeepSHAP / GradientSHAP Path integral Integrated Gradients Expected Gradients Guided IG / Blur IG Integrated Hessians Gradient / backpropagation Saliency / SmoothGrad DeepLIFT LRP / FullGrad Grad-CAMâ Perturbation / surrogate Occlusion LIME RISE Extremal perturbations value function vâ(S)v(S), Sec. IV path Îł and baseline xâ˛x , Sec. V derivative / conservation rule, Sec. VI perturbation distribution p_ z, Sec. VII Figure 4: Taxonomy of local additive feature attribution methods used throughout this survey, with the mathematical object that each family primarily manipulates (italics): a cooperative-game value function, a path integral with its baseline, a backpropagation-style derivative or conservation rule, or a black-box perturbation distribution. Methods marked â (CAM family) are gradient-based but produce spatial attributions through an intermediate-layer projection. Section VIII places all four families in a common axiom-by-method comparison. This section presents the main comparison table of the survey. Having surveyed Shapley, path-based, gradient/backpropagation, and perturbation methods, we now place them in a single mathematical frame. We compare them along five axes: axioms satisfied, value function choice, baseline distribution, computational complexity, and known equivalences/reductions. The centerpiece is the axiom-by-method matrix (Table IX) recording which axioms each method satisfies. The matrix illustrates that many of the fieldâs central disagreements, examined as failure modes in Section X, arise from incompatible axiom subsets, not from implementation defects. VIII-A Axes of Comparison We compare methods along the following axes: Axioms satisfied. Catalogued in Section I; collected in Table IX. Value function choice. Interventional, conditional, or single-reference (Section I). Baseline / path / perturbation. The specific reference, path, or perturbation distribution used. Computational complexity. In flops, forward/backward passes, or samples, as a function of d and any approximation parameter. Differentiability assumption. Required for gradient/path methods, not for Shapley or perturbation methods. Causal assumptions. Whether the methodâs output admits a causal interpretation, and under what conditions. Stability / scalability. Sensitivity to small input changes; behaviour as dââdââ. VIII-B Axiom Satisfaction Matrix Table IX cross-references methods (rows) against axioms (columns). A â means the method satisfies the axiom unconditionally under the assumptions of its original paper; an â marks a conditional claim (specific architecture, sampling limit, propagation rule, or method variant); a blank means the axiom is not satisfied in general; N/A marks axioms that are inapplicable to the methodâs output type. The matrix records mathematical axioms only. Empirical evidence about sanity-check behaviour is implementation- and architecture-dependent and lies outside the axiomatic claims. It is reported separately in Table X, following the critique that sanity checks are properties of an implementation tested on a dataset, not of a method formula [4, 94]. TABLE IX: Axiom satisfaction matrix. â= satisfied unconditionally under the stated original assumptions; â = conditional (architecture, limit, rule, or method variant); blank = not satisfied; N/A = inapplicable to the methodâs output type. Abbreviations: Comp = completeness; S(a)/S(b) = Sensitivity-(a)/(b); I = implementation invariance; Cons = consistency; Mono = monotonicity; Sym = symmetry-preservation; Lin = linearity; Cont = continuity in x. Per-cell justifications are in Appendix A. The entries refer to canonical formulations unless otherwise stated; software implementations may differ. Method Comp S(a) S(b) I Cons Mono Sym Lin Cont Shapley family Exact Shapley [82] â â â â â â â â â KernelSHAP [62] â â â â â â â â â TreeSHAP [61] â â â â â â â â â DeepSHAP [62] â â â â â â â â GradientSHAP [33] â â â â â â â â Path family Integrated Gradients [93] â â â â â â â Expected Gradients [33] â â â â â â â Guided IG [52] â â â â â â Blur IG [100] â â â â â â Integrated Hessians [48] â â â â â â â Gradient & backpropagation family Saliency [84] â â â â Grad Ă Input â â â â â â SmoothGrad [87] â â â â Guided BP [88] â Deconvnet [102] â DeepLIFT [83] â â â â â â â LRP [9] â â â â â â FullGrad [89] â â â â â â Grad-CAM [80] â â â â â HiResCAM [32] â â â â â â Score-CAM [97] â â â â â Perturbation family Occlusion [102] â â â â â â LIME [73] â â â â â â Meaningful Pert. [34] â â â â RISE [71] â â â â â â Boundary cases (not local additive on âdR^d) Anchors [74] N/A â â â N/A N/A SAGE [22] (global) N/A N/A â â â â â N/A TABLE X: Empirical-evidence companion to Table IX. Each row reports published empirical evidence that the method passes (â) or fails (Ă) the model-randomization sanity check [4] and the data-randomization sanity check [4]. Entries marked â are partial or method-variant-dependent (e.g., Grad-CAM variants differ). Blank evidence cells make no randomization-check claim for that method; blank source cells are included only for matrix completeness. Sanity-check behaviour depends on implementation, architecture, and dataset; entries here are not axiomatic claims. Method Model-rand. evidence Data-rand. evidence Source(s) Saliency â â [4] Grad Ă Input â â [4] SmoothGrad â â [4] Guided BP Ă Ă [4] Deconvnet Ă [4] DeepLIFT (Rescale) â [4] LRP-Ͼξ â [4] Integrated Gradients â â [4] Grad-CAM family â [4, 32] HiResCAM â [32] KernelSHAP / TreeSHAP LIME RISE source reports perturbation metrics, not randomization checks VIII-C Complexity Landscape Table XI collects the asymptotic computational costs. The methods span seven orders of magnitude: from a single backward pass for raw saliency to Oâ(2d)O(2^d) for exact Shapley. TABLE XI: Computational complexity of representative methods. netnet denotes one forward+backward pass through the model. T,L,DT,L,D are the number of trees, leaves, and tree depth; K is the number of path steps; M the number of samples; N the number of SmoothGrad noise draws. Method Complexity Saliency / Grad Ă Input Oâ(net)O(net) SmoothGrad / VarGrad Oâ(Nâ net)O(N¡net) DeepLIFT, LRP, FullGrad Oâ(net)O(net) Grad-CAM family Oâ(net)O(net) Score-CAM Oâ(Kâ netfwd)O(K¡net_fwd) Integrated Gradients Oâ(Kâ net)O(K¡net) Expected Gradients / GradientSHAP Oâ(Kâ Mâ net)O(K¡ M¡net) Integrated Hessians Oâ(K2â net)O(K^2¡net) TreeSHAP Oâ(TâLâD2)O(TLD^2) KernelSHAP (sampled) Oâ(Mâ dâ netfwd)O(M¡ d¡net_fwd) Exact Shapley Oâ(2dâ netfwd)O(2^d¡net_fwd) LIME (sampled) Oâ(Mâ netfwd)O(M¡net_fwd) RISE Oâ(Nâ netfwd)O(N¡net_fwd) Anchors Oâ(Mâ netfwd)O(M¡net_fwd) (rule search) VIII-D Equivalence and Reduction Theorems Several methods that appear distinct in the literature are mathematically equivalent or related by explicit reductions; the differences in notation between the original papers tend to hide this. Result 8 (KernelSHAP and exact Shapley, Lundberg & Lee [62], Thm. 2). Assumptions: the interventional value function (1) is used; the weighted least-squares regression in (5) is solved exactly on the full mask distribution (no sampling). Under these assumptions, the KernelSHAP solution equals the exact Shapley value (6). Finite-sample bias and variance are characterized in [21, 20]. Result 9 (DeepLIFT-Rescale and IG, Ancona et al. [6], §3). Assumptions: linear layers and monotone elementwise nonlinearities; no skip connections, concatenations, or non-monotone operators; single fixed baseline. Under these assumptions, DeepLIFT with the Rescale rule and Integrated Gradients (with KââKââ) yield equivalent attributions in the small-input-increment limit. The reduction is sensitive to architectural assumptions and does not extend automatically to transformer or residual networks. Result 10 (Grad-CAM as class-conditional gradient projection; reformulation, not reduction). Assumptions: CNN with global average pooling followed by a linear classification head. Under these assumptions, the Grad-CAM map of (28) is a rewriting (not a reduction) of a class-conditional spatial average of the final-feature-map gradient [80], and coincides with the original CAM [104] on the GAP-then-linear architecture. We label this a reformulation because the two computations yield the same output by construction; it is not a reduction between distinct methods. Result 11 (LRP-Ͼξ and gradient-Ă-input, Ancona et al. [6], §3.2â3.3). Assumptions: deep ReLU network with no bias terms; LRP with the Ͼξ-rule applied uniformly to every layer; the input is viewed as the difference xâx- 0 from the zero baseline. Under these assumptions, LRP-Ͼξ reduces to gradient Ă input, which is the K=1K=1 Riemann approximation of Integrated Gradients with xâ˛=x = 0. The reduction fails when biases are present, when LRP composition (different rules at different layers) is used, or when the baseline is non-zero. These reductions matter in practice: they imply that empirical disagreements between methods should be tracked back to differences in the axioms (which method ignores which), the value function (which notion of feature absence is in play), or the path (which trajectory through input space is integrated), not to differences in mathematical sophistication. VIII-E Taxonomy Table Table XII situates the methods on three orthogonal axes: their mathematical object (the primary computation), whether they are local or global, and whether they are model-specific or model-agnostic. TABLE XII: Taxonomy of attribution methods. Family Math. object Local/Global Model-specific Input type Output Shapley / cooperative game coalition value function local (mostly) no any vector ââd ^d Path-based line integral of gradient local yes (diff.) continuous vector ââd ^d Gradient / backprop 1st-order derivative / rules local yes (diff.) continuous vector / heatmap CAM family spatial gradient projection local yes (CNN) image 2D heatmap Perturbation / occlusion black-box query local no any vector / heatmap Surrogate (LIME) local linear regression local no any vector Rule-based (Anchors) rule predicate local no any rule set SAGE global Shapley over loss global no any vector ââd ^d VIII-F The Central Disagreement, Geometrically A useful way to summarize the comparison is geometric. Each attribution method is a function ÎŚ:âąĂââd ĂX ^d (37) mapping a (model, input) pair to an attribution vector, parameterized by a choice of value function v, baseline distribution xâ˛D_x , path Îł, and perturbation distribution p_ z. Two methods that disagree on Ďâ(x) Ď(x) disagree because they live at different points in =value fnĂbaselinesĂpathsĂperturbations.C=\value fn\Ă\baselines\Ă\paths\Ă\perturbations\. The empirical question âwhich method should I use?â is therefore not a question about ÎŚ but about which neighbourhood of C matches the userâs intended question. We close this section, and the methods half of the survey, with the following statement. Principle 1. A disagreement between attribution methods ÎŚ1 _1 and ÎŚ2 _2 on a prediction fâ(x)f(x) is informative about the model only to the extent that ÎŚ1 _1 and ÎŚ2 _2 share a value function, baseline, path, and perturbation distribution. Disagreement under different choices reflects the choices, not the model. Principle 1 summarizes the comparison: attribution choices are modelling choices, and explanation comparison should hold those choices constant. IX Evaluation Theory and Metrics Evaluating an attribution method is harder than producing one. There is no ground-truth attribution against which to measure, and the purposes for which attributions are used, including model debugging, user trust, regulatory compliance, and scientific discovery, impose different evaluation criteria. This section surveys the principal evaluation frameworks that have emerged. We organize them around the property each metric purports to measure: faithfulness to the model, stability under input changes, alignment with ground-truth annotations, and the validity of the metric itself. Input x,model ffAttributionĎâ(f,x) Ď(f,x)Perturb /mask / retrainModelresponseFaithfulness /stability metricrankobservedchangepredictedchange Figure 5: Generic evaluation pipeline for attribution methods. The attribution Ďâ(f,x) Ď(f,x) ranks or selects features; the model is queried under perturbations or retrainings indexed by the ranking; the observed change in the modelâs response is compared against the change the attribution predicts, producing a faithfulness or stability metric. Section IX catalogues the instantiations. IX-A Faithfulness: Does the Attribution Reflect the Model? Faithfulness measures whether the attribution reports what the model actually does, as opposed to what a human would do given the same input. Faithfulness has been operationalized in several inequivalent ways. Perturbation response. Remove the top-k features by attribution magnitude and measure the drop in prediction. Used in [78] and codified by DeYoung et al. [27] as âcomprehensivenessâ (drop when top features are removed) and âsufficiencyâ (drop when only top features are retained). Functional faithfulness. âiĎiâfâ(x)âfâ(xâ˛) _i _iâ f(x)-f(x ), i.e. completeness as an evaluation metric distinct from the axiom. Methods that satisfy completeness pass trivially; methods that do not are penalized. Causal faithfulness. Defined by Jacovi and Goldberg [46], who distinguish faithfulness from plausibility and argue that the two are often conflated in NLP attribution. IX-B Infidelity and Sensitivity Yeh et al. [101] proposed two metric families that have become standard. Definition 1 (Infidelity). For an attribution Ď Ď and a perturbation distribution ÎźI _I, INFDâ(Ď,f,x)=IâźÎźIâ[(Iâ¤âĎâ(fâ(x)âfâ(xâI)))2].INFD( Ď,f,x)=E_I _I\! [ (I Ď-(f(x)-f(x-I)) )^2 ]. (38) Infidelity measures the mean-squared error between (i) the attributionâs prediction of how the model output changes under perturbation I and (i) the actual change. Methods that satisfy completeness in expectation under ÎźI _I minimize (38). Different choices of ÎźI _I yield different metrics: a Gaussian ÎźI _I tests robustness to noise; a sparse ÎźI _I tests feature removal. Definition 2 (Max-Sensitivity). SENSmaxâ(Ď,f,x,r)=supâyâxââ¤râĎâ(f,y)âĎâ(f,x)â.SENS_max( Ď,f,x,r)= _\|y-x\|⤠r\| Ď(f,y)- Ď(f,x)\|. Max-sensitivity bounds how much the attribution can change when the input is perturbed within radius r. It is closely related to the Lipschitz stability axiom (Axiom 16); a Lipschitz attribution has bounded max-sensitivity for every r. Yeh et al. also characterize the optimum: for a fixed perturbation distribution ÎźI _I, the attribution minimizing infidelity (38) is the generalized least-squares solution Ďâ=IâźÎźIâ[IâIâ¤]â1âIâźÎźIâ[Iâ(fâ(x)âfâ(xâI))], Ď =E_I _I\! [I ]^-1\,E_I _I\! [I\, (f(x)-f(x-I) ) ], (39) a smoothed, kernel-weighted gradient of f around x. They further show that kernel smoothing of a given attribution lowers its max-sensitivity and, under the conditions stated in their paper, does not worsen its infidelity; empirically it often improves both. Stability and fidelity are therefore not strictly opposed. The real tension is between fidelity to f at the point x and fidelity averaged over the neighbourhood that ÎźI _I defines. IX-C ROAR and KAR: Retraining-Based Faithfulness Hooker et al. [45] argued that perturbation-response metrics like (30) are confounded by distribution shift: removing features may simply push the input off-manifold, in which case a prediction drop may reflect manifold violation and fail to measure feature importance. ROAR (RemOve and Retrain) addresses this by retraining the model after feature removal. Given an attribution method, mask the top-k%k\% of features, train a fresh model on the masked data, and compare its test accuracy to a baseline that masks k%k\% randomly. A faithful attribution should produce a substantially larger accuracy drop than random. KAR (Keep And Retrain) is the dual: retain only the top-k%k\% features. A faithful attribution should still allow the model to learn. ROAR/KAR is expensive because each evaluation point requires a full retraining, and so has been applied at small scale, but it is one of the more defensible faithfulness criteria in the literature because it explicitly controls for distribution shift. IX-D Sanity Checks Adebayo et al. [4] proposed the model-randomization and data-randomization sanity checks: ⢠Model-randomization: replace the modelâs weights (layer-by-layer or in cascade) with random values; the attribution should change substantially. An attribution that does not is not explaining the model. ⢠Data-randomization: train the model on data with permuted labels; the attribution should change. An attribution that does not is not sensitive to what the model has learned. These checks are necessary for an attribution method to be called an explanation, but they are not sufficient: a method can pass both and still be uninformative for downstream tasks. The most-discussed empirical finding from Adebayo et al. is the failure of guided backpropagation and deconvolution under the model-randomization check. We incorporated this finding directly into Table X. IX-E Insertion / Deletion Curves For image attribution, Petsiuk et al. [71] introduced two curves: Insertion. Start from a baseline (e.g., a blurred image) and add features in decreasing order of attribution magnitude. Plot the prediction as a function of the number of features inserted; a faithful attribution should rise quickly. Deletion. Start from x and delete features in decreasing order of attribution. The prediction should fall quickly. The area under the resulting curves (AUC-Ins, AUC-Del) summarizes the methodâs performance; Figure 6 sketches the geometry. The metric has the virtue of being entirely model-internal: no ground-truth annotation is required. fraction of features k/dk/dfcf_c011insertionAUC-Insdeletionrandom order Figure 6: Insertion and deletion curves [71]. Features are inserted into a baseline input (solid) or deleted from x (dashed) in decreasing order of attribution magnitude. A faithful attribution makes the prediction rise quickly under insertion and fall quickly under deletion, relative to a random ordering (dotted); the shaded area is AUC-Ins. Both curves depend on the baseline used for insertion and on the replacement value used for deletion, which is the same baseline-sensitivity caveat as in Section X. IX-F Localization Metrics When ground-truth annotations are available, such as bounding boxes for object detection or rationales for text classification, the attribution can be compared directly to them: Pointing game. Does the maximum-attribution pixel lie within the ground-truth bounding box? (Standard in CAM evaluation.) IoU. Treat the top-k attributions as a predicted mask and compute intersection-over-union with the ground-truth mask. Token F1 / IOU. In NLP, treat selected tokens as a predicted rationale and compare against human rationales [27]. Localization metrics conflate plausibility (alignment with human annotations) with faithfulness (alignment with model behaviour). A method can have perfect IoU while being unfaithful (e.g., if the model relies on features outside the ground-truth box but the attribution defers to the human annotation). IX-G Using Attribution During Training Another evaluation route is to make explanations actionable during training. Ross, Hughes, and Doshi-Velez [76] penalize input gradients at annotated irrelevant dimensions, training models that are accurate and at the same time constrained away from known spurious reasons. Rieger et al. [75] extend this idea through contextual-decomposition explanation penalization, including feature interactions. These methods do not replace post-hoc faithfulness tests; they show that an attribution method can be validated by whether its signal can guide a model away from documented confounders under a stated training objective. IX-H Human-Grounded Evaluation Doshi-Velez and Kim [31] categorized evaluations into three tiers: functionally-grounded (model-internal, no human required), human-grounded (simplified tasks with non-expert users), and application-grounded (deployed tasks with domain experts). Most attribution evaluations have been functionally-grounded; application-grounded evaluations remain rare and are often the most decision-relevant. DeYoung et al. [27] introduced the ERASER benchmark, which provides human rationales for several NLP tasks and treats attribution evaluation as a token-selection problem. The benchmark exposes a striking gap: even attribution methods that score well on functional metrics often fail to identify the tokens that humans annotate as evidence. For model debugging in text classification, Bastings et al. [13] propose a complementary shortcut-based protocol: inject known lexical shortcuts, verify that the trained model uses them, and evaluate whether salience methods rank the shortcut tokens near the top. This protocol is important because it provides a controlled ground truth for model reliance, something a human rationale alone cannot supply. IX-I Sanity Checks on the Metrics A second-order concern, raised by Tomsett et al. [94], is that the evaluation metrics themselves can fail sanity checks. They showed that several popular faithfulness metrics correlate weakly across methods. A method ranked highest by one metric is often ranked lowest by another, and some metrics depend strongly on hyperparameters in ways the original authors did not document. The methodological implication is that no single metric should be treated as authoritative; a method should be evaluated against multiple, ideally diverse, metrics, with the disagreements among them reported. IX-J Summary Table XIII summarizes the principal evaluation metrics. No single metric dominates: faithfulness, stability, and plausibility are different quantities, and a method can score well on one while failing another. What the metrics jointly require is stated as items R8âR9 of the reporting checklist (Section XII). TABLE XIII: Evaluation metrics for attribution methods. âRequiredâ is what the metric needs beyond the attribution and the model; âLimitationsâ lists the principal known failure modes. Metric What it measures Required Principal limitation Completeness check functional faithfulness none trivial for methods satisfying it as axiom Comprehensiveness / Sufficiency [27] top-k perturbation response none off-manifold distortion Infidelity [101] MSE between attribution and Îâf f noise distribution ÎźI _I depends on choice of ÎźI _I Max-sensitivity [101] local stability of Ď Ď radius r infidelity/sensitivity trade-off ROAR / KAR [45] retraining-based faithfulness retraining budget very expensive Sanity (model-rand.) [4] non-degeneracy w.r.t. model weight randomization necessary, not sufficient Sanity (data-rand.) [4] non-degeneracy w.r.t. data label permutation necessary, not sufficient Insertion / Deletion AUC [71] ordering quality of attribution baseline (e.g., blur) strong baseline dependence Pointing game / IoU alignment with bounding box ground-truth box conflates plausibility, faithfulness ERASER token F1/IoU [27] alignment with human rationale human rationale limited to text tasks Shortcut protocol [13] recovery of known model shortcut controlled shortcut data task construction must match debugging hypothesis Application study downstream utility human users + task expensive, low statistical power X Failure Modes and Theoretical Gaps The standard analyses of attribution failure modes by Adebayo et al. [4], Kindermans et al. [54], Ghorbani et al. [39], Slack et al. [86], and Kumar et al. [56] present these failures as discoveries about specific methods. We recast them as mathematical phenomena tied to specific choices of value function, baseline, path, or perturbation distribution. This recasting clarifies which failures are artefacts of method design and which are intrinsic to the attribution problem itself. X-A Gradient Saturation and Shattering Let f be differentiable. Saturation occurs when âf/âxiâ0â f/â x_iâ 0 in a region around x, even though f is locally sensitive to xix_i in a broader sense. The canonical example is a ReLU network in which a unitâs gradient is zero on its inactive side; on long inputs the cumulative effect is that important features receive zero saliency. For example, take fâ(x)=1âeâ5âxf(x)=1-e^-5x at x=1x=1 with baseline xâ˛=0x =0. The prediction difference is Îâf=1âeâ5â0.993 f=1-e^-5â 0.993, yet the local gradient is fâ˛â(1)=5âeâ5â0.034f (1)=5e^-5â 0.034: saliency and gradient Ă input both report the feature as nearly irrelevant. The path integral repairs this by averaging the gradient over the whole trajectory and avoiding reliance on the saturated endpoint: IG1âĄ(1;0)=(1â0)ââŤ015âeâ5âÎąâÎą=1âeâ5=Îâf.IG_1(1;0)=(1-0) _0^15e^-5Îą\,dÎą=1-e^-5= f. (40) Figure 7 plots the same effect for a sigmoidal model along its attribution path. Saturation is a property of the local geometry of f, so any purely local method, including raw saliency, gradient Ă input, and the K=1K=1 Riemann approximation of IG, is vulnerable. The mathematical fix is to integrate: IG with sufficiently large K reduces saturation effects by averaging over the path and recovering completeness as the numerical integral converges. Path methods reduce this failure mode, although the result still depends on the baseline and path. saturatedregionpath position Îą 011fâ(Îłâ(Îą))f(Îł(Îą))dâÎąâfâ(Îłâ(Îą)) ddÎąf(Îł(Îą)) (scaled)x Figure 7: Gradient saturation along an attribution path. For a sigmoidal f, the gradient (dashed) is concentrated in the middle of the path and nearly zero at the input x itself (shaded band). Saliency evaluated at x misses the feature; the path integral (15) accumulates the mid-path gradient mass and recovers the full prediction difference, as in the worked example (40). Gradient shattering is a stronger failure: in deep networks, gradients become high-frequency and approximately independent across inputs, with white-noise-like behaviour inconsistent with a smooth function. The effect grows with depth and is a known property of deep networks [11]. SmoothGrad addresses shattering by averaging gradients over a noise neighbourhood, effectively replacing f with a smoothed version f~Ď f_Ď. The trade-off is between locality (small Ď) and stability (large Ď). X-B Baseline Sensitivity For any method that requires a baseline, including IG, EG, DeepLIFT, LRP, and Shapley methods with vintv^int, the attribution depends on the baseline choice. Sturmfels et al. [91] showed how strong this dependence can be: for the same model and input, IG with a zero baseline and IG with a mean baseline can yield attributions that disagree on the sign for most features. This is not strictly speaking a failure because a baseline is a modelling choice and different baselines answer different questions. However, the empirical literature often treats it as one. We propose the following classification: Remark 2. A baseline-dependence âfailureâ is meaningful only if (i) the researcher does not state which baseline they used, or (i) the researcher claims their attribution is canonical when it depends on a hyperparameter. Otherwise, baseline dependence is the correct behaviour. X-C Correlated and Dependent Features Real features are correlated, and most attribution methods are defined assuming independence at one or more points. The interventional value function (1) produces inputs with implausibly extreme features when applied to highly correlated coordinates; the conditional value function (2) requires estimating d-dimensional conditional distributions, which is intractable except in special cases. The mathematical statement of the problem, paraphrasing Theorem 3 of Janzing et al. [49], is: Result 12 (Paraphrase of Janzing et al. [49], Thm. 3). For some f and some pairs of features Xi,XjX_i,X_j that are perfectly correlated under pXp_X, the Shapley value ĎiShap _i^Shap under vintv^int and under vcondv^cond can differ by an arbitrary factor. The precise sufficient conditions are in the cited paper. The implication is sharp: a researcher who reports âthe Shapley value of feature iâ without specifying the value function has reported an underdetermined quantity. Approximate solutions: Aas et al. [1] estimate vcondv^cond via a multivariate Gaussian or copula model; Frye et al. [37] use a learned generative model to sample on-manifold coalition completions; Janzing et al. [49] argue that the question itself should be reformulated as a causal one. X-D Off-Manifold Perturbations Many evaluations perturb the input by setting features to zero, blurring them, or replacing them with samples from a marginal distribution. The resulting inputs typically lie far from the data manifold, and the modelâs prediction at these inputs is undefined in a deeper sense: the model was never trained on such inputs and its behaviour there is extrapolation, not interpretation. Mathematically, off-manifold perturbation breaks the implicit assumption of all perturbation-based methods that f behaves locally well outside suppâ(pX)supp(p_X). Hooker et al. [45] showed that the apparent âfaithfulnessâ of an attribution measured by perturbation response can be entirely explained by the modelâs degradation on off-manifold inputs, not by the attributionâs identification of important features. The principled fix is to constrain perturbations to the data manifold: generative-model perturbations (FIDO-CA [17], Schulz et al. [79]) and on-manifold Shapley (Frye et al. [37]) both implement this. The cost is an additional model whose own validity must be verified. X-E Causal Misinterpretation A pervasive failure lies in interpretation: reading attribution as a causal effect. The following statements have all appeared in published applied papers: ⢠âFeature i caused the prediction fâ(x)>0f(x)>0.â ⢠âIncreasing feature i by 1 unit would change the prediction by Ďi _i units.â ⢠âFeature i is more important than feature j for the underlying phenomenon.â None of these is supported by any standard attribution method without strong additional assumptions about the data-generating process. Attribution methods compute associations (in the loose sense of sensitivity, marginal contribution, or conditional expectation), typically with respect to a model trained by maximum likelihood. They do not compute causal effects on the world. Janzing et al. [49] give a constructive bridge: a class of attribution methods that admit a causal interpretation under the assumption that the model is a fair approximation of the conditional expectation and that the user has correctly specified the causal graph. The conditions are strong; the broader point is that attribution-as-causation requires a separate, explicit causal model. X-F Adversarial Manipulation of Explanations Adversarial attacks on explanations show that the attribution map Ďâ(f,x) Ď(f,x) can be made nearly arbitrary by a small, nearly imperceptible perturbation δ that preserves the prediction fâ(x+δ)âfâ(x)f(x+δ)â f(x). Result 13 (Adversarial manipulability of explanations; Ghorbani et al. [39] and Dombrowski et al. [30]). For standard gradient-based attributions on ReLU networks, including saliency, gradient Ă input, and Grad-CAM, the cited papers construct, for typical inputs x, a perturbation δ of small norm âδââ¤Ďľ\|δ\|â¤Îľ such that the prediction is nearly preserved, |fâ(x+δ)âfâ(x)||f(x+δ)-f(x)| is at most Oâ(Ďľ)O(Îľ), while the attribution map changes by a substantial amount âĎâ(x+δ)âĎâ(x)ââĽC\| Ď(x+δ)- Ď(x)\|⼠C, where C can be made arbitrarily large. The constructions exploit the piecewise-linear geometry of ReLU networks; we state this as a summary grounded in the cited constructions and do not supply a self-contained proof. The geometric intuition due to Dombrowski et al. [30] is that for ReLU networks, the prediction is piecewise linear but the attribution is piecewise constant; small movements across pieces change the attribution discontinuously while leaving the prediction nearly unchanged. The mathematical fix is to enforce Lipschitz stability (Axiom 16), which SmoothGrad approximates and expected-gradient methods enforce in expectation. Slack et al. [86] extended adversarial attacks to LIME and SHAP: a model can be trained that behaves discriminatorily on the data distribution but produces non-discriminatory explanations under LIME or KernelSHAP. The attack exploits the off-manifold queries that LIME and KernelSHAP make: a model that is anomalous on those queries can hide its real behaviour. The fix is the same as for the off-manifold problem: constrain queries to the data manifold. X-G Disagreement Among Methods A practical concern, often the first one users encounter, is that different attribution methods disagree on the same prediction. Krishna et al. [55] formalized this as the disagreement problem, measured across six methods and four benchmarks; their results show median rank correlations below 0.50.5 between methods that are widely treated as interchangeable. Bilodeau et al. [15] provided the complementary theoretical statement: no attribution method can simultaneously satisfy a small list of intuitive properties (linearity, completeness, and a faithfulness-style criterion), so disagreement is not an empirical accident of any particular method. Section VIII (Principle 1) established that disagreement is informative only when the methods share a value function, baseline, path, and perturbation distribution. The most common form of reported disagreement, for example between vanilla saliency and KernelSHAP, fails this condition: the methods compute different quantities, so it would be strange if they agreed. The implication for practice is that disagreement calls for further investigation before it supports any conclusion. âIG and SHAP disagreeâ is not the same finding as âIG with the same baseline, value function, and number of samples produces a different ordering on different runsâ; only the latter indicates a method-level instability. We propose a short protocol for investigating disagreement: (i) verify that both methods use the same scalar output fcâ(x)f_c(x) and the same feature granularity; (i) compare value functions (interventional vs. conditional) and baselines; (i) quantify approximation error in each method against a higher-budget reference; (iv) only if (i)â(i) match, report the methods as substantively disagreeing. X-H Specification-to-Failure Map Table XIV summarizes the methodological lesson of the preceding subsections: many failures are traceable to an unstated choice in the attribution specification. The table is a diagnostic map for deciding what a paper must report before an attribution claim can be interpreted. TABLE XIV: Hidden choices and the failure modes they commonly induce. Hidden choice Failure mode Example Off-manifold perturbation Implausible explanations occlusion, SHAP, LIME Baseline choice Baseline sensitivity IG, DeepLIFT Gradient saturation Missing important features saliency, low-K IG Rule-modified backpropagation Sanity-check failure guided backprop Sampling distribution Instability / variance LIME, KernelSHAP, RISE X-I Summary: Failure Modes Are Specifications of Assumptions Table XV summarizes the failure modes. The unifying view is that each failure occurs when an attribution method is asked a question it was not designed to answer. Saturation occurs when a method is asked about a region in which the gradient is locally uninformative. Baseline sensitivity occurs when a method is asked for a result without a baseline being specified. Off-manifold failure occurs when a method is asked to extrapolate. Causal misinterpretation occurs when a method is asked for a causal answer it does not provide. Each failure has a structural fix: path integration for saturation, explicit baselines for sensitivity, manifold-aware perturbations for off-manifold, and explicit causal modelling for causal claims. None of those fixes is method-agnostic. The implication is the following: an attribution paper claiming general superiority over prior methods should identify which failure mode it addresses and what assumptions or trade-offs the mitigation introduces. TABLE XV: Failure modes of attribution methods, the methods principally affected, the formal cause, and the structural mitigation. Failure mode Methods affected Formal cause Mitigation Gradient saturation saliency, âfĂxâ fĂ x, low-K IG âfâ0â fâ 0 on the path KââKââ, DeepLIFT-Rescale Gradient shattering many gradient methods on deep nets high-frequency âfâ f SmoothGrad (Ď>0Ď>0) Baseline sensitivity IG, EG, DeepLIFT, LRP, vintv^int Shapley answer depends on xâ˛x report xâ˛x ; use expected baseline Correlated features vintv^int Shapley, KernelSHAP off-manifold coalition completions conditional Shapley / manifold-aware Off-manifold perturbations LIME, KernelSHAP, RISE, occlusion f undefined outside suppâ(pX)supp(p_X) generative perturbations Causal misinterpretation associational methods associational â causal causal Shapley + DAG Adversarial manipulation non-Lipschitz methods Result 13 Lipschitz / smoothed attribution Method disagreement cross-family comparisons differing value fn, path, xâ˛D_x hold method-parameters constant XI Applications and Model Families The methods of Sections IVâVII differ in their suitability across model families and input modalities. This section is brief by design: our focus is the mathematical analysis, not the application landscape. We summarize, for each major model family, which method choices are mathematically appropriate, which are empirically established, and which open problems remain. XI-A Tabular Models Tabular models, including gradient-boosted trees [36], random forests [16], generalized linear models, and deep tabular networks, are the original domain of SHAP and remain its strongest fit. TreeSHAP [61] gives exact Shapley values in polynomial time for tree ensembles, removing the principal obstacle (exponential cost) that applies to other Shapley implementations. For tabular data the choice between interventional and conditional value functions is unusually consequential: real tabular features are often highly correlated (e.g., age, tenure, and account balance in credit scoring), and the two value functions can produce contradictory explanations. The current recommendation, following [49, 92], is to report interventional TreeSHAP as the default and to discuss conditional TreeSHAP when correlation between features is structural and meaningful (e.g., when two columns are functionally related). LIME and KernelSHAP remain competitive for non-tree tabular models. XI-B Convolutional Image Models For CNNs, the attribution literature has converged on a small set of methods: Grad-CAM and HiResCAM for coarse spatial attribution; Integrated Gradients and Expected Gradients for pixel-level attribution; LRP and DeepLIFT for backpropagation-style attribution; RISE and Score-CAM for gradient-free attribution; and XRAI for region-aggregated attribution. The key methodological points specific to images are: ⢠The zero baseline is almost always wrong for image inputs; a blurred or mean-image baseline is more appropriate [91]. ⢠Pixel-level attribution is often dominated by high-frequency artefacts; SmoothGrad smoothing or XRAI region aggregation are usually required for human-readable maps. ⢠Adebayo et al. [4] sanity-checks should be run for any image attribution method; failure of the sanity check implies the heatmap does not depend on what the model has learned. XI-C Transformers and NLP For transformer language models, attention rollout [2] and transformer relevance propagation [19] are the principal model-aware attribution methods. For NLP classification specifically, the ERASER benchmark [27] provides the standard evaluation; LIME, IG (with a token-embedding baseline), DeepLIFT, and LRP all have NLP-specific implementations [7]. The chief NLP-specific difficulty is that token-level attribution ignores compositional structure: a sentenceâs meaning depends on the interaction of tokens, not their individual contributions. Hierarchical methods [85], contextual decomposition [68], and feature-interaction attribution [96] address this directly. The debate between attention-as-explanation [47, 81, 99] has not been settled but has clarified that attention is at best a partial input-feature attribution: it captures which tokens the model attends to but not why those attentions matter for the prediction. Shortcut-based evaluation [13] is especially useful in NLP because many relevant confounders are lexical or template-like, and therefore can be injected, verified, and evaluated under controlled data modifications. For large language models, attribution is largely an open problem (Section XIV): the same input may produce different generations under different sampling, gradient computations are expensive at trillion-parameter scale, and the meaningful unit of explanation may be sub-token, token, span, or document. Token-level attribution for large language models inherits the same issues surveyed here but adds complications from discrete inputs, subword tokenization, prompt dependence, retrieval context, and generation-time decoding. We treat these as extensions within the local attribution problem and outside the surveyâs separate targets. XI-D Graph Neural Networks For GNNs, attribution must respect graph structure: features include node attributes, edge attributes, and the graph topology itself. Extensions of LIME (GraphLIME), SHAP (GraphSHAP), and Integrated Gradients (GNN-IG) to graphs have been developed in dedicated lines of work, which we do not survey here. The mathematical points relevant to our axiomatic framing are: (i) the value function must condition on both the graph adjacency structure and the node features; (i) permutation symmetries induced by graph isomorphism interact with the symmetry axiom; (i) message-passing computations admit a layer-wise relevance propagation extension analogous to LRP for feed-forward networks. XI-E Biomedical and High-Stakes Domains In biomedical applications, the failure modes of Section X are not abstract concerns. A clinical decision-support system that attributes credit to a feature for spurious reasons (gradient saturation, off-manifold perturbation, adversarial weight choice during training) can produce confident, visually convincing explanations that are systematically misleading. The methodological consensus emerging in this domain prefers: ⢠Methods that satisfy completeness and sensitivity-(a) as axioms (IG, SHAP variants, LRP, DeepLIFT) over methods that satisfy them only approximately; ⢠Methods with documented sanity-check behaviour; ⢠Multiple methods reported in parallel, with disagreement analyzed under Principle 1 of Section VIII; ⢠Training-time explanation constraints when domain knowledge identifies invalid reasons for a prediction [76, 75]; ⢠Application-grounded evaluation [31] involving domain experts in addition to functional metrics. These are the same principles that emerge in any high-stakes decision domain: legal, financial, hiring. The mathematical machinery of axioms, value functions, and failure modes is the common ground; domain expertise determines which axioms are required for which decisions. XI-F Application-Domain Matrix Table XVI summarizes which methods are preferred across application domains, the principal cautions, and standard evaluation conventions. TABLE XVI: Application-domain matrix for attribution methods. Domain Preferred methods Principal cautions Standard evaluation Tabular TreeSHAP, KernelSHAP, LIME feature correlation; value-function choice ROAR, insertion/deletion Image (CNN) IG, EG, Grad-CAM, HiResCAM, XRAI, RISE baseline choice; sanity checks; off-manifold Insertion/Deletion AUC, sanity checks, IoU Text (transformer) IG (token), DeepLIFT, transformer LRP, attention rollout attention vs. attribution; compositional structure ERASER token F1/IoU, comprehensiveness/sufficiency Graph (GNN) GNN-IG, GraphSHAP, GNN-LRP graph structure as feature; permutation symmetry ablation by edge / node subset Biomedical TreeSHAP / IG with documented baselines off-manifold; adversarial trojans; multi-method reporting ROAR + application-grounded Sequence Contextual decomposition, hierarchical, LRP-RNN long-range dependence; vanishing gradients ERASER-style + perturbation LLM (open problem) N/A cost; sampling randomness; meaningful unit (see Section XIV) XII A Proposed Reporting Checklist for Attribution Studies Attribution methods encode modelling assumptions, and comparison or use of an attribution requires those assumptions to be stated. This section converts that observation into a proposed reporting checklist: a paper or applied study that uses an attribution method should report enough information that another researcher can (i) reproduce the attribution from the same inputs and model, (i) place the result correctly on the axiom matrix (Table IX) and the failure-mode table (Table XV), and (i) decide whether the reported attribution is appropriate for the userâs question. The checklist has ten items, organized into three blocks: specification of the question, specification of the method, and specification of the evidence. TABLE XVII: Ten-item reporting checklist for local additive attribution. The items specify the minimum information needed to reproduce and interpret an attribution claim. Item What to report Why it matters R1 Model, training context, and scalar output fixes the target of attribution R2 Feature granularity, grouping, and display transform prevents comparing pixels, tokens, words, and groups as if identical R3 Baseline or reference distribution defines the comparison point R4 Value function or perturbation distribution defines feature absence and locality R5 Path, coalition strategy, or sampling design determines approximation and axioms R6 Axioms satisfied and not satisfied states the mathematical guarantees R7 Approximation budget, stochasticity, seeds, and uncertainty makes sampled attributions reproducible R8 Sanity-check behaviour tests model and data dependence R9 Faithfulness or stability metric evaluates the attribution claim R10 Known failure modes for the setting prevents overinterpretation XII-A Block A: Specifying the Question R1. Model and scalar output. State (i) the model f being explained, including its architecture, training data, and whether it is treated as a black box or as a differentiable function; (i) the scalar output fcâ(x)f_c(x) being attributed, including the class logit, post-softmax probability, regression output, loss, or other quantity; and (i) the input x and whether the attribution is local (per-input) or aggregated. R2. Feature definition, grouping, and display transform. State what counts as a feature: raw pixels, super-pixels, tokens, sub-tokens, structured groups, one-hot categories, or graph nodes/edges. Feature granularity is part of the attribution problem definition: attributions over pixels, superpixels, tokens, words, and domain variables are not directly comparable unless the grouping map is stated. Also state whether attributions are signed, absolute-valued, normalized, clipped, smoothed, thresholded, aggregated, or recolored before visualization. R3. Reference / baseline distribution. State the baseline xâ˛x or reference distribution xâ˛D_x used. If a single xâ˛x is used, state its choice (zero, mean, blurred image, paraphrase, etc.). If an expected baseline is used, state the distribution pXp_X from which references are drawn and the number of samples. XII-B Block B: Specifying the Method R4. Value function or perturbation distribution. State explicitly which value function is used: interventional ((1)), conditional ((2)), single-reference ((3)), or other. For perturbation methods, state the perturbation distribution p_ z. This item fixes the meaning of feature absence and locality. R5. Path or coalition strategy. For path methods, state the path Îł (straight-line, adaptive, scale-space) and the number of Riemann steps K. For coalition methods, state the sampling strategy (KernelSHAP weighting, permutation sampling, TreeSHAP recursion) and the sample budget M. R6. Axioms satisfied / not satisfied. State which axioms from Section I the chosen method satisfies, under the chosen value function and baseline. Where a known reduction or equivalence applies (e.g., Result 8, Result 9), cite it. R7. Computational approximation and stochasticity. State the approximation error budget and the corresponding sample size, integration steps, or network-pass count. For sampling-based methods, report the number of samples, random seeds, variance estimates, and convergence diagnostics where feasible. XII-C Block C: Specifying the Evidence R8. Sanity-check results. Report the outcome of at least one of the model-randomization and data-randomization sanity checks of Adebayo et al. [4]. A method that is largely insensitive to model parameters should not, by itself, be used to support strong claims about model-specific reasoning. R9. Faithfulness or stability metric. Report at least one metric of faithfulness (infidelity (38), comprehensiveness/sufficiency [27], ROAR/KAR accuracy drop [45], or insertion/deletion AUC [71]) and at least one metric of stability (max-sensitivity, empirical Lipschitz, or VarGrad-style variance). We recommend the Quantus toolkit [44] as a reference implementation supporting reproducible computation of these metrics. R10. Known failure modes that apply. Map the chosen method onto Table XV and state which failure modes are plausibly active for the chosen setting. For example, an interventional Shapley method on a tabular dataset with correlated features should explicitly acknowledge the correlated-features failure mode and the value-function ambiguity. XII-D Compliance Examples We illustrate the checklist with two short compliance examples; the purpose is to show how short the report can be when the methodological choices are unambiguous. Example 1: TreeSHAP on a credit-risk model. R1: gradient-boosted tree ensemble, target = probability of default for input record x; local attribution. R2: features = the 14 tabular columns of the credit dataset. R3: interventional baseline distribution = empirical training distribution, 1000 reference samples. R4âR5: interventional value function with TreeSHAP exact recursion. R6: satisfies efficiency, symmetry, dummy, additivity, local accuracy, missingness, consistency (see Table IX). R7: exact computation; no sampling error. R8: model-randomization sanity check passes (cosine similarity <0.1<0.1 to attribution under random weights). R9: infidelity = 0.0120.012 at Gaussian noise Ď=0.1Ď=0.1. R10: features âincomeâ and âdebt-to-income ratioâ are correlated (Ď=0.71Ď=0.71); interventional Shapley under-attributes their joint effect (Table XV, row âcorrelated featuresâ). Example 2: Integrated Gradients on an image classifier. R1: ResNet-50 ImageNet classifier, target = pre-softmax logit for the predicted class. R2: pixel-level attribution. R3: blurred-input baseline (Gaussian blur, Ď=30Ď=30 pixels). R4âR5: single-reference value function; straight-line path with K=200K=200 Riemann steps. R6: satisfies sensitivity-(a), sensitivity-(b), implementation invariance, completeness, linearity, symmetry (Result 4). R7: integration error empirically below 1%1\% on test images relative to K=2000K=2000 reference. R8: sanity-check pass under cascading randomization. R9: max-sensitivity = 0.180.18 at r=0.02r=0.02; insertion AUC = 0.740.74. R10: gradient saturation possible on inactive ReLU paths (Table XV); off-manifold artefacts at intermediate Îą on the blur path. XII-E Summary The proposed compact, checkable checklist converts the assumptions encoded by attribution methods into a reproducible reporting practice. The checklist does not restrict which attribution methods may be used. It specifies the information needed for attribution results to support the conclusions drawn from them. It is offered to authors and reviewers as a proposal, not as a community-ratified standard. XIII Scope and Boundary Conditions The principal scope boundaries and their rationale keep the mathematical object of study fixed: local additive attribution operators for predictive models. XIII-A Explicit Out-of-Scope The following topics are adjacent to the survey but outside its main scope because they manipulate mathematical objects distinct from the local additive attribution operator. ⢠Mechanistic interpretability. Circuit-level explanations of neural network internals (induction heads, sparse autoencoders, polysemantic neurons) target a different question: âwhat does the model compute?â This differs from the question âwhat features mattered for this prediction?â These methods do not produce a per-feature attribution vector and are not comparable to the methods we survey within the frame of Sections IâI. ⢠Concept-based and concept-bottleneck explanations. TCAV [53], network dissection [14], and concept-bottleneck models replace input features with named human concepts. They are treated briefly in Section I as the natural contrast to feature attribution; a full survey of the concept literature is a separate project. ⢠Counterfactual and example-based explanations. The literature on counterfactual explanations (âwhat would have to change about x for the prediction to flip?â) and on example-based explanations (influence functions, prototypes, training-data attribution) addresses questions complementary to but distinct from local additive attribution. We mention FIDO-CA and counterfactual generators (Section VII) where they intersect the attribution literature; we do not survey the broader counterfactual research programme. ⢠Global interpretability and rule extraction. Methods that produce a global summary of a black-box model (decision lists, partial dependence plots, ALE plots, rule extraction [57]) target a different deliverable than per-prediction attribution. SAGE [22] is the closest member of the Shapley family to a global method and is covered for that reason; the broader global-interpretation literature is surveyed in [77, 59]. ⢠Causal explanation methods. Methods grounded in structural causal models or Pearlâs do-calculus (causal Shapley [49], do-attribution) are discussed in their connection to interventional value functions (Section IV) and as an open problem (Section XIV), but a full survey of causal attribution would require a separate exposition of causal-graph machinery beyond the scope of this paper. ⢠Comprehensive LLM-specific attribution. The LLM attribution literature has grown rapidly in 2023â2025; we discuss its open status in Section XIV. A full treatment would need to cover token-level vs. span-level attribution for autoregressive generation, in-context-learning attribution, sparse attention probes, and emerging mechanistic-interpretability tools for transformers. XIII-B Topics Treated Briefly The following topics receive a section or subsection in this paper and have dedicated literatures of their own. ⢠Graph neural network attribution (Section XI): permutation symmetry, edge-attribution, and graph-conditional value functions deserve a separate treatment. ⢠Biomedical and high-stakes-domain applications (Section XI): the domain-specific evaluation criteria are surveyed lightly; we focus on the general methodological consensus. ⢠Evaluation in NLP (Section IX): the ERASER [27] benchmark is the standard reference, and the broader debate on plausibility vs. faithfulness in NLP is ongoing; we summarize but do not extend the faithfulness debate. XIII-C Interpretive Boundaries of the Taxonomy The taxonomy is a conceptual comparison and does not provide an empirical ranking. The scoring of prior surveys in Table I depends on judgment about what counts as substantive coverage; for this reason, the scoring rule is stated explicitly and ambiguous cases are assigned in favour of the prior survey. Several attribution methods also have implementation-dependent variants. An axiom entry may hold for the canonical formulation but not for every software implementation or propagation rule. Claims about sanity checks, infidelity, robustness, and method disagreement are therefore synthesized as reported properties of the cited studies, with implementation-dependent cases marked separately from mathematical axioms. XIII-D Threats to Validity of the Taxonomy Four limitations qualify the conclusions that should be drawn from the taxonomy and its tables. ⢠Corpus limitation. The corpus prioritizes methodological and axiomatic relevance over exhaustive bibliometric coverage; counts and coverage claims should be read accordingly. ⢠Axiom-classification uncertainty. Several methods exist in multiple variants (LRP rules, DeepLIFT rules, CAM layers), so matrix entries depend on which variant and implementation is taken as canonical; the conditional marks and Appendix A record these dependencies, but borderline judgments remain. ⢠Domain dependence. Attribution behaviour differs across images, text, tabular data, graphs, and biological sequences; properties observed in one modality do not automatically transfer. ⢠Evaluation non-equivalence. Faithfulness, stability, sanity checks, and human usefulness measure different quantities (Section IX); the taxonomy does not assume that any one of them subsumes the others. XIII-E Methodological Boundaries The survey is structured and taxonomy-driven. Its corpus construction prioritizes papers that define attribution operators, establish axioms, evaluate attribution behaviour, or document failure modes. ⢠The axiom-by-method matrix (Table IX) encodes the authorsâ reading of each methodâs axiomatic profile; where a methodâs behaviour depends on architectural details or on a choice of rule (LRP rules, DeepLIFT rules), we mark the entry conditionally and refer to the cited paper. ⢠Several reductions reported in Section VIII as âResultsâ are paraphrases of theorems in the cited papers, with assumptions made explicit. We have tried to flag every assumption that is critical to the reduction. ⢠Empirical results we cite (e.g., the sanity-check failure of guided backpropagation) are reported as such, not as theorems. Where quantitative claims are made, they reflect the original authorsâ experimental setting, not a meta-analysis. XIV Open Problems and Future Directions We close with six directions in which the methods of Sections IVâVII are incomplete. Each is stated as a question that current theory answers partially or not at all. XIV-A Causal Attribution The clearest open problem is the rigorous integration of attribution methods with causal modelling. Janzing et al. [49] provided the first systematic formulation, but the conditions under which Shapley-style attribution admits a causal interpretation remain restrictive. Open questions include: ⢠How does attribution interact with confounding? An attribution method may attribute credit to feature i that, in the underlying causal graph, is mediated by an unobserved confounder. ⢠Can attribution methods be characterized as estimating some causal estimand, such as average treatment effect, conditional ATE, or individualized treatment effect, under the modelâs implicit causal assumptions? ⢠For interventional Shapley, what is the relationship between the do-operator and the value function? Recent work suggests a tight correspondence, but a full theorem remains to be stated. XIV-B Correlated Features and Manifold-Aware Attribution The conditional vs. interventional value-function debate (Section IV) remains live. Manifold-aware methods [37, 1, 79] constrain perturbations or coalition completions to the data manifold, but they require an additional generative model whose own faithfulness is now a question. Two open problems: ⢠How should the additional generative model be trained, evaluated, and validated? Standard generative-model evaluation (FID, likelihood) is not obviously appropriate for an attribution sub-routine. ⢠Is there an axiomatic characterization of on-manifold Shapley analogous to Result 1? The Lundstrom-Razaviyayn-style characterization [63] of IG does not yet have a manifold-aware analogue. XIV-C Uncertainty in Explanations Attribution methods are deterministic functions of (f,x)(f,x). Yet the model itself is a sample from a learning algorithm, and the attribution should plausibly carry uncertainty propagated from the modelâs training process. Existing approaches include Bayesian-network attribution, ensemble disagreement, and posterior-distribution attribution, but the principled framework remains open. The natural question is whether the attribution should be a distribution over âdR^d, and if so what its marginals should mean. XIV-D Attribution for Large Language Models LLM attribution is the most active open area in XAI today. Four specific difficulties: ⢠What is the prediction? LLMs produce sequences, not scalars. Attribution targets include per-token logits, sequence probabilities, and aggregate metrics like answer correctness; the choice matters. ⢠What is the feature? Sub-tokens, tokens, spans, sentences, documents, system prompts, and retrieved context are all candidate units. Hierarchical methods [85] extend cleanly here but require deciding the granularity in advance. ⢠Computational cost. A trillion-parameter forward pass makes IG-style K-step path integration prohibitively expensive; gradient methods are similarly costly at scale. ⢠In-context learning attribution. For few-shot LLM predictions, the explanation may need to attribute credit to specific demonstrations in the context. This is a new problem class without an obvious analogue in the methods we have surveyed. XIV-E Rigorous Benchmarks and Human-Aligned Axioms The benchmarks of Section IX are largely functionally-grounded and often disagree with one another. Two directions seem productive: ⢠Application-grounded benchmarks. ERASER [27] is one such; analogues are needed for vision (clinical image cohorts), biology (gene-perturbation prediction), and structured prediction (graph classification). The principal obstacle is the cost of expert annotation. ⢠Human-aligned axioms. The axioms surveyed in Section I are mathematically motivated. A complementary line of work would establish which axioms humans actually endorse for an explanation, and whether the satisfaction of those axioms predicts downstream utility. XIV-F Explanation Robustness Result 13 establishes that gradient-based attributions are intrinsically fragile. The constructive question is how to build attributions that are stable by design. Two directions: ⢠Lipschitz-constrained attribution. Enforce Lipschitz stability (Axiom 16) directly, either by smoothing the model or by smoothing the attribution. SmoothGrad does the latter in expectation; principled methods that enforce a Lipschitz constant explicitly would be a substantial advance. ⢠Certifiable attribution. Analogous to certifiable robustness for predictions, the question is whether one can certify that the attribution map has a bounded change under any âδââ¤Ďľ\|δ\|â¤Îľ. Existing approaches based on randomized smoothing provide a partial answer, but tight bounds remain open. XIV-G Conclusion We began this survey by arguing that the mathematical objects manipulated by feature-attribution methods provide the appropriate basis for comparison. The intervening sections have substantiated that argument: the methods discussed here, from exact Shapley to Grad-CAM to LIME, can be situated in a common frame defined by a value function, a baseline distribution, a path or perturbation distribution, and a conservation rule. The axiomatic analysis (Section I, Table IX) and the failure-mode formalization (Section X, Table XV) make precise the conditions under which each methodâs attribution is well-defined, the questions it can and cannot answer, and the assumptions it requires of the underlying data and model. The thesis stated in Section I, restated here as a closing principle: There is no assumption-free feature-attribution method. Every local additive attribution method defines feature importance through choices about value functions, references, paths, perturbation distributions, or conservation rules. Trustworthy use therefore requires reporting a heatmap or ranking together with the assumptions under which the attribution was computed and interpreted. For practice, this means that both method papers and applied papers should state the assumptions behind the reported attribution: a new attribution method should state its value function, baseline, path, and conservation rule, and an applied paper that reports attributions without specifying these choices has reported an underdetermined quantity. The resulting catalogue supports a more explicit style of attribution reporting: method comparisons should name the mathematical choices they hold fixed, and applied studies should state the choices on which their attributions depend. Reproducibility Statement Reproducible artefacts. (i) The axiom-by-method matrix (Table IX) is derivable from the per-cell justifications in Appendix A; each cell references either the original method paper, an axiomatic characterization, or a stated conditional assumption. A reader can verify each cell against the cited paper without further implementation. (i) The empirical-evidence companion (Table X) cites the published sources for each sanity-check outcome; these external empirical claims were not generated in this survey. (i) The checklist crosswalk in Appendix B records the source literature that motivates each reporting item and shows how the items apply across method families. Supplementary materials. The supplementary material includes a machine-readable reporting checklist, a review-corpus summary, an axiom-matrix CSV, and a CSV recording the scoring rationale for Table I. These materials provide stable artefacts for checking the taxonomy, the proposed reporting checklist, and the survey-positioning table. Acknowledgments The authors thank the maintainers of the open-source attribution libraries (Captum, SHAP, iNNvestigate, Quantus, Grad-CAM++) whose implementations underlie much of the empirical XAI literature. Appendix A Per-Cell Justification of the Axiom Matrix This appendix supplies, for each row of Table IX, a short justification or citation for each non-trivial axiom claim. We use the following abbreviations: Cmp (Completeness), Sa/Sb (Sensitivity-(a)/(b)), I (Implementation Invariance), Cns (Consistency), Mono (Monotonicity), Sym (Symmetry-preservation), Lin (Linearity), and Cont (Continuity in x). Entries in the matrix marked â are justified here unconditionally; entries marked â are conditional, with the condition stated. Shapley Family Exact Shapley. Cmp, Sym, dummy and Lin are Axioms 1â4; satisfied by construction via [82]. Cns is a direct consequence of monotone marginal contributions. I follows from v depending only on the inputâoutput function of f. Cont in x holds for any continuous value function v. KernelSHAP [62]. In the limit of infinite samples, equals exact Shapley (Result 8); inherits its axioms. Cns and Mono are conditional (â ) because finite-sample KernelSHAP can exhibit variance-induced violations, documented in [21, 20]. TreeSHAP [61]. Inherits all Shapley axioms exactly; Cns is provably satisfied (Theorem 1 of [61]). Cont is conditional because TreeSHAP attributions are piecewise constant in x. DeepSHAP [62]. Cmp is approximate (one forward+backward pass; full Shapley requires multiple). Sa and Cns are conditional (â ) and I is conditional because both depend on the DeepLIFT propagation rule used (Rescale vs. RevealCancel). GradientSHAP [33]. Equation (12): the inner expectation over Îą is an IG operator, the outer expectation over xâ˛x is a sampling step. Inherits Cmp, Sa, Sb, I, Lin, Sym from IG. Cns conditional on sampling adequacy. Path Family Integrated Gradients [93]. Cmp by fundamental theorem of calculus (Result 4); Sa, Sb, I, Lin, Sym by Theorem 1 of the cited paper. Cont follows from the integrability of the gradient. Expected Gradients [33]. Same as IG, with the baseline replaced by an expectation. Cmp, Sa, Sb, I, Lin, Sym all preserved by linearity of the expectation operator. Guided IG [52]. Cmp preserved because the integration is still over a path. I is conditional because the path depends on the gradient magnitude, which can break strict invariance under network reparameterization. Sym is not preserved. Blur IG [100]. Cmp on the scale-space path; inherits I, Lin from IG. Sym fails because the path is not permutation-equivariant. Integrated Hessians [48]. Cmp over feature pairs; inherits I, Sym, Lin from the IG construction. Sa, Sb at the interaction level. Gradient and Backpropagation Family Saliency [84]. Sb by definition (Ďi=0 _i=0 if âf/âxiâĄ0â f/â x_i⥠0). I by invariance of partial derivatives. Cmp fails (sum of gradients â Îâfâ f). Cont conditional (â ) due to ReLU non-smoothness. Gradient Ă Input. Cmp is satisfied only when f is linear (â ). I by invariance of âfâ f. Sa conditional on nonzero gradient. SmoothGrad [87]. Smoothing preserves I and Sb; gives Cont by construction (averaging Gaussian kernel). Cmp not satisfied. Guided BP [88] / Deconvnet [102]. Do not satisfy I because the rule-modified backward pass is non-standard. Their model-randomization behaviour is reported separately in Table X. DeepLIFT [83]. Cmp by summation-to-delta (Axiom 15). Sa, Sb, Sym, Lin by construction. I conditional on Rescale rule and architecture. LRP [9]. Cmp by conservation (Axiom 14). Sa, Sb conditional on the propagation rule. I conditional on standard rules and absence of bias terms. FullGrad [89]. Cmp by explicit bias accounting. Sa, Sb, I, Cont by the smooth differentiability of the total gradient plus biases. Grad-CAM [80]. Cmp not satisfied at the pixel level. I conditional on the architecture having global average pooling. HiResCAM [32]. Cmp at the layer level (â ). Sa, Sb by direct gradient application (no pooling). I is conditional (â ): the map is defined on a chosen internal layer, and functionally equivalent networks need not share internal representations. Score-CAM [97]. Does not require gradients, but the weights are computed from selected activation maps of a target layer, so I is conditional (â ): strict implementation invariance holds only for methods that depend on inputâoutput queries alone. Sa by construction (perturbation-based). Perturbation Family Occlusion [102]. Sa for any patch with nonzero marginal contribution. I by black-box queries. Sym by permutation-equivariance of patch sliding. Cmp fails (single coalition, not exhaustive). LIME [73]. Cmp conditional (â ) on the kernel; satisfied only when the kernel is the Shapley kernel (10). I by black-box queries. Cont by smoothness of the local linear fit. Anchors [74]. Anchors output rules, not real-valued attributions; Cmp, Lin, Cont are N/A. Sa and Sb apply analogously (rule-based). Meaningful Perturbations [34]. Sa via optimization objective. I by black-box queries. Cont conditional on the regularization Îť. RISE [71]. Sa by construction. Sym by mask distribution being permutation-symmetric. I by black-box queries. Notes on Conditional Entries The â marks in Table IX signal that the axiom holds under specific conditions: choice of rule (LRP, DeepLIFT), architectural restrictions (Grad-CAM family, DeepLIFT), sample adequacy (KernelSHAP, GradientSHAP), or method variant (CAM family). Readers extracting machine-readable claims from the matrix should consult the per-method paragraph above for the precise qualification. Interpretation of the Matrix The matrix entries above summarize mathematical claims and implementation-dependent empirical statuses reported in the cited papers. Where a methodâs behaviour depends on a configuration (e.g., LRP-Ͼξ vs. LRP-composition), the matrix records the most commonly used configuration as the default and the alternative as a â . Appendix B Checklist Crosswalk for the Analytic Corpus This appendix connects the proposed reporting checklist of Section XII to the source literature used throughout the survey. The goal is to show how each checklist item follows from established attribution practice, axiomatic characterization, or documented failure modes. The crosswalk also provides a compact guide for applying the checklist across method families. Item-Level Source Anchors Table XVIII records, for each item R1âR10, the specification target, representative source anchors in the analytic corpus, and the itemâs role in the checklist. Each anchor is a paper that either motivated the item or documented the consequence of leaving it unreported. TABLE XVIII: Source anchors for the reporting checklist across the analytic corpus. Each row identifies representative papers that motivate the corresponding checklist item. Item Specification target Representative source anchors Role in the standard R1 Model and scalar output Saliency [84], Grad-CAM [80], ERASER [27] Fixes whether the attribution explains a logit, probability, loss, or task-specific score. R2 Feature unit and display transform LIME [73], TCAV [53], hierarchical attribution [85], transformer attribution [19] Prevents direct comparison of pixels, superpixels, tokens, spans, concepts, and grouped variables. R3 Baseline or reference distribution Integrated Gradients [93], DeepLIFT [83], baseline studies [91] Defines the counterfactual reference point against which feature contributions are measured. R4 Value function or perturbation distribution Shapley values [82], SHAP [62], conditional Shapley [1], manifold Shapley [37], causal attribution [49], QII [25] Specifies feature absence, locality, and whether correlated features are treated interventionally or conditionally. R5 Path, coalition strategy, or sampling design IG [93], Guided IG [52], KernelSHAP [62, 21], RISE [71] Determines numerical approximation, sampling variance, and the applicable axiom guarantees. R6 Axioms satisfied and not satisfied Shapley [82], Banzhaf [12], Aumann-Shapley [8], IG axioms [93], IG characterization [63], impossibility results [15] States which mathematical guarantees the attribution can legitimately support. R7 Approximation budget and stochasticity SmoothGrad [87], KernelSHAP analysis [21], Shapley algorithms [20], RISE [71] Makes sampled, noisy, or discretized attributions reproducible. R8 Sanity-check behaviour Sanity checks [4], sanity-check extensions [94], HiResCAM [32] Tests whether the attribution depends on learned model parameters and training labels. R9 Faithfulness or stability metric Infidelity [101], ROAR [45], ERASER [27], RISE metrics [71], robustness [5], shortcut tests [13], Quantus [44] Connects attribution values to model behaviour under controlled perturbation or stability tests. R10 Known failure modes Input invariance [54], fragility [39], manipulation [30], fooling attacks [86], disagreement [55], explanation constraints [76, 75] Links each attribution claim to the failure modes most relevant to its method family and data setting. Cross-Family Reporting Profile Table XIX groups the same ten items by method family, separating the items that are usually necessary to make the attribution mathematically specified from the items that support its empirical use. TABLE XIX: Checklist emphasis by method family. âPrimaryâ items are usually necessary to make the attribution mathematically specified; âevidenceâ items support empirical use of the attribution. Method family Primary specification items Evidence items Representative sources Shapley and value-function methods R1âR5, especially R4 for conditional/interventional choice R6, R7, R9, R10 [82, 90, 25, 62, 61, 21, 1, 37, 49] Path-integral methods R1âR3 and R5 for baseline and path R6, R7, R8, R10 [93, 33, 52, 100, 63] Gradient and backpropagation methods R1, R2, R6 and the propagation rule R8, R9, R10 [84, 88, 9, 83, 6, 89] CAM-style spatial methods R1, R2 and target layer / projection rule R8, R9, R10 [104, 80, 18, 97, 32, 103] Perturbation and surrogate methods R1, R2, R4 and R5 for perturbation design R7, R9, R10 [73, 74, 34, 71, 72, 17, 79] Transformer and NLP attribution R1 and R2 for scalar target and token/span unit R8, R9, R10 [7, 68, 81, 47, 2, 19, 27, 46, 13] Use of the Crosswalk The crosswalk gives the reporting checklist a literature-backed interpretation. For a new attribution study, the relevant method family in Table XIX identifies the items that must be specified before the attribution is mathematically defined. The source anchors in Table XVIII then identify the literature that explains why each item matters. For example, a KernelSHAP study requires the scalar output (R1), feature grouping (R2), the interventional or conditional value function (R4), the coalition sampling design (R5), the approximation budget (R7), and the correlated-feature failure mode (R10). An Integrated Gradients study requires the scalar output (R1), baseline (R3), path and integration budget (R5, R7), axioms used (R6), sanity-check behaviour (R8), and baseline/path sensitivity (R10). References [1] K. Aas, M. Jullum, and A. Løland (2021) Explaining individual predictions when features are dependent: more accurate approximations to Shapley values. Artificial Intelligence 298. Cited by: TABLE XVIII, TABLE XIX, §X-C, §XIV-B, item Conditional value., §IV-H. [2] S. Abnar and W. Zuidema (2020) Quantifying attention flow in transformers. In Proc. ACL, Cited by: TABLE XIX, §XI-C, §VI-H. [3] A. Adadi and M. Berrada (2018) Peeking inside the black-box: a survey on explainable artificial intelligence. IEEE Access 6, p. 52138â52160. Cited by: item 1, §I-D, TABLE I. [4] J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim (2018) Sanity checks for saliency maps. In Proc. NeurIPS, Vol. 31, p. 9525â9536. Cited by: TABLE XVIII, item 2, §I, §X, 3rd item, §XII-C, §VI-B, TABLE VIII, TABLE VIII, §VIII-B, TABLE X, TABLE X, TABLE X, TABLE X, TABLE X, TABLE X, TABLE X, TABLE X, TABLE X, TABLE X, §IX-D, TABLE XIII, TABLE XIII, Result 6. [5] D. Alvarez-Melis and T. S. Jaakkola (2018) On the robustness of interpretability methods. In ICML Workshop on Human Interpretability, Cited by: TABLE XVIII, Axiom 16. [6] M. Ancona, E. Ceolini, C. Ăztireli, and M. Gross (2018) Towards better understanding of gradient-based attribution methods for deep neural networks. In Proc. ICLR, Cited by: TABLE XIX, item 3, item 2, §V-C, Result 11, Result 7, Result 9. [7] L. Arras, G. Montavon, K. MĂźller, and W. Samek (2017) Explaining recurrent neural network predictions in sentiment analysis. In EMNLP WASSA Workshop, Cited by: TABLE XIX, §XI-C. [8] R. J. Aumann and L. S. Shapley (1974) Values of non-atomic games. Princeton University Press. Cited by: TABLE XVIII, §I-C. [9] S. Bach, A. Binder, G. Montavon, F. Klauschen, K. MĂźller, and W. Samek (2015) On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE 10 (7), p. e0130140. Cited by: Appendix A, TABLE XIX, §I, §I-E, §VI-E, TABLE VIII, TABLE IX, Axiom 14. [10] D. Baehrens, T. Schroeter, S. Harmeling, M. Kawanabe, K. Hansen, and K. MĂźller (2010) How to explain individual classification decisions. Journal of Machine Learning Research 11, p. 1803â1831. Cited by: §VI-A. [11] D. Balduzzi, M. Frean, L. Leary, J. P. Lewis, K. W. Ma, and B. McWilliams (2017) The shattered gradients problem: if resnets are the answer, then what is the question?. In Proc. ICML, Proceedings of Machine Learning Research, Vol. 70, p. 342â350. Cited by: §X-A. [12] J. F. Banzhaf (1965) Weighted voting doesnât work: a mathematical analysis. Rutgers Law Review 19, p. 317â343. Cited by: TABLE XVIII, §I-C. [13] J. Bastings, S. Ebert, P. Zablotskaia, A. Sandholm, and K. Filippova (2022) âWill you find these shortcuts?â a protocol for evaluating the faithfulness of input salience methods for text classification. arXiv:2111.07367. Cited by: TABLE XVIII, TABLE XIX, §XI-C, §IX-H, TABLE XIII. [14] D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba (2017) Network dissection: quantifying interpretability of deep visual representations. In Proc. CVPR, Cited by: §I-C, 2nd item. [15] B. Bilodeau, N. Jaques, P. W. Koh, and B. Kim (2024) Impossibility theorems for feature attribution. Proceedings of the National Academy of Sciences 121 (2). Note: Art. no. e2319169121 External Links: Document Cited by: TABLE XVIII, item 2, §I, §X-G. [16] L. Breiman (2001) Random forests. Machine Learning 45 (1), p. 5â32. Cited by: §XI-A, §VII-B. [17] C. Chang, E. Creager, A. Goldenberg, and D. Duvenaud (2019) Explaining image classifiers by counterfactual generation. In Proc. ICLR, Cited by: TABLE XIX, §X-D, §I-D, §VII-F. [18] A. Chattopadhyay, A. Sarkar, P. Howlader, and V. N. Balasubramanian (2018) Grad-CAM++: improved visual explanations for deep convolutional networks. In Proc. WACV, p. 839â847. External Links: Document Cited by: TABLE XIX, §VI-G, TABLE VIII. [19] H. Chefer, S. Gur, and L. Wolf (2021) Transformer interpretability beyond attention visualization. In Proc. CVPR, Cited by: TABLE XVIII, TABLE XIX, §XI-C, §VI-H. [20] H. Chen, I. C. Covert, S. M. Lundberg, and S. Lee (2023) Algorithms to estimate Shapley value feature attributions. Nature Machine Intelligence 5, p. 590â601. Cited by: Appendix A, TABLE XVIII, §IV-C, §IV-I, Result 8. [21] I. Covert and S. Lee (2021) Improving KernelSHAP: practical Shapley value estimation using linear regression. In Proc. AISTATS, Cited by: Appendix A, TABLE XVIII, TABLE XVIII, TABLE XIX, item 3, item 2, item 3, §IV-C, TABLE VI, Result 8. [22] I. Covert, S. Lundberg, and S. Lee (2020) Understanding global feature contributions with additive importance measures. In Proc. NeurIPS, Cited by: 4th item, §IV-F, TABLE VI, TABLE IX. [23] A. Cremades, S. Hoyas, and R. Vinuesa (2024) Additive-feature-attribution methods: a review on explainable artificial intelligence for fluid dynamics and heat transfer. arXiv:2409.11992. Cited by: item 1, §I-D, TABLE I. [24] P. Dabkowski and Y. Gal (2017) Real time image saliency for black box classifiers. In Proc. NeurIPS, Cited by: §VII-G. [25] A. Datta, S. Sen, and Y. Zick (2016) Algorithmic transparency via quantitative input influence: theory and experiments with learning systems. In Proc. IEEE Symposium on Security and Privacy, p. 598â617. External Links: Document Cited by: TABLE XVIII, TABLE XIX, §IV-B. [26] S. Desai and H. G. Ramaswamy (2020) Ablation-CAM: visual explanations for deep convolutional network via gradient-free localization. In Proc. WACV, Cited by: §VI-G. [27] J. DeYoung, S. Jain, N. F. Rajani, E. Lehman, C. Xiong, R. Socher, and B. C. Wallace (2020) ERASER: a benchmark to evaluate rationalized NLP models. In Proc. ACL, Cited by: TABLE XVIII, TABLE XVIII, TABLE XIX, §XI-C, §XII-C, 3rd item, 1st item, item Perturbation response., item Token F1 / IOU., §IX-H, TABLE XIII, TABLE XIII. [28] K. Dhamdhere, A. Agarwal, and M. Sundararajan (2020) The Shapley-Taylor interaction index. In Proc. ICML, Cited by: §IV-G, §IV-G, TABLE VI, §V-D. [29] K. Dhamdhere, M. Sundararajan, and Q. Yan (2019) How important is a neuron?. In Proc. ICLR, Cited by: §V-E, TABLE VII. [30] A. Dombrowski, M. Alber, C. Anders, M. Ackermann, K. MĂźller, and P. Kessel (2019) Explanations can be manipulated and geometry is to blame. In Proc. NeurIPS, Vol. 32, p. 13567â13578. Cited by: TABLE XVIII, §I, §X-F, §I-F, §VI, Result 13. [31] F. Doshi-Velez and B. Kim (2017) Towards a rigorous science of interpretable machine learning. arXiv:1702.08608. Cited by: §I-B, 5th item, §IX-H. [32] R. L. Draelos and L. Carin (2020) Use HiResCAM instead of Grad-CAM for faithful explanations of CNNs. arXiv:2011.08891. Cited by: Appendix A, TABLE XVIII, TABLE XIX, §VI-G, TABLE VIII, TABLE X, TABLE X, TABLE IX. [33] G. Erion, J. D. Janizek, P. Sturmfels, S. M. Lundberg, and S. Lee (2021) Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature Machine Intelligence 3, p. 620â631. Cited by: Appendix A, Appendix A, TABLE XIX, 3rd item, §IV-E, §IV-E, TABLE VI, 3rd item, TABLE VII, TABLE IX, TABLE IX. [34] R. C. Fong and A. Vedaldi (2017) Interpretable explanations of black boxes by meaningful perturbation. In Proc. ICCV, Cited by: Appendix A, TABLE XIX, §VII-D, TABLE IX. [35] R. Fong, M. Patrick, and A. Vedaldi (2019) Understanding deep networks via extremal perturbations and smooth masks. In Proc. ICCV, Cited by: §VII-D. [36] J. H. Friedman (2001) Greedy function approximation: a gradient boosting machine. The Annals of Statistics 29 (5), p. 1189â1232. Cited by: §XI-A, §VII-B. [37] C. Frye, D. de Mijolla, T. Begley, L. Cowton, M. Stanley, and I. Feige (2021) Shapley explainability on the data manifold. In Proc. ICLR, Cited by: TABLE XVIII, TABLE XIX, §I-B, §X-C, §X-D, §XIV-B, item Conditional value., §IV-H. [38] D. Fryer, I. StrĂźmke, and H. Nguyen (2021) Shapley values for feature selection: the good, the bad, and the axioms. arXiv:2102.10936. Cited by: §IV-F. [39] A. Ghorbani, A. Abid, and J. Zou (2019) Interpretation of neural networks is fragile. In Proc. AAAI, Cited by: TABLE XVIII, item 2, §I, §X, §I-F, §VI, Result 13. [40] A. Ghorbani and J. Zou (2020) Neuron Shapley: discovering the responsible neurons. In Proc. NeurIPS, Cited by: §IV-G. [41] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal (2018) Explaining explanations: an overview of interpretability of machine learning. In Proc. IEEE DSAA, Cited by: item 1, §I-D, TABLE I. [42] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi (2018) A survey of methods for explaining black box models. ACM Computing Surveys 51 (5). Cited by: item 1, §I-D, TABLE I. [43] T. Han, S. Srinivas, and H. Lakkaraju (2022) Which explanation should i choose? A function approximation perspective to characterizing post hoc explanations. In Proc. NeurIPS, Cited by: item 2. [44] A. HedstrĂśm, L. Weber, D. Bareeva, D. Krakowczyk, F. Motzkus, W. Samek, S. Lapuschkin, and M. M.-C. HĂśhne (2023) Quantus: an explainable AI toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research 24 (34), p. 1â11. Cited by: TABLE XVIII, §XII-C. [45] S. Hooker, D. Erhan, P. Kindermans, and B. Kim (2019) A benchmark for interpretability methods in deep neural networks. In Proc. NeurIPS, Vol. 32. Cited by: TABLE XVIII, §I, §X-D, §XII-C, §IX-C, TABLE XIII. [46] A. Jacovi and Y. Goldberg (2020) Towards faithfully interpretable NLP systems: how should we define and evaluate faithfulness?. In Proc. ACL, Cited by: TABLE XIX, item Causal faithfulness.. [47] S. Jain and B. C. Wallace (2019) Attention is not explanation. In Proc. NAACL, Cited by: TABLE XIX, §XI-C, §VI-H. [48] J. D. Janizek, P. Sturmfels, and S. Lee (2021) Explaining explanations: axiomatic feature interactions for deep networks. Journal of Machine Learning Research 22 (104), p. 1â54. Cited by: Appendix A, §IV-G, §V-D, TABLE VII, TABLE IX, Result 5. [49] D. Janzing, L. Minorics, and P. BlĂśbaum (2020) Feature relevance quantification in explainable AI: a causal problem. In Proc. AISTATS, Proceedings of Machine Learning Research, Vol. 108, p. 2907â2916. Cited by: TABLE XVIII, TABLE XIX, §I-B, §X-C, §X-C, §X-E, §XI-A, 5th item, §XIV-A, §I-C, §IV-D, §IV-H, Result 12. [50] P. Jiang, C. Zhang, Q. Hou, M. Cheng, and Y. Wei (2021) LayerCAM: exploring hierarchical class activation maps for localization. IEEE Trans. Image Processing 30, p. 5875â5888. Cited by: §VI-G. [51] A. Kapishnikov, T. Bolukbasi, F. ViĂŠgas, and M. Terry (2019) XRAI: better attributions through regions. In Proc. ICCV, Cited by: §V-C, TABLE VII. [52] A. Kapishnikov, S. Venugopalan, B. Avci, B. Wedin, M. Terry, and T. Bolukbasi (2021) Guided integrated gradients: an adaptive path method for removing noise. In Proc. CVPR, Cited by: Appendix A, TABLE XVIII, TABLE XIX, §I-D, §V-C, TABLE VII, TABLE IX. [53] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. ViĂŠgas, and R. Sayres (2018) Interpretability beyond feature attribution: quantitative testing with concept activation vectors (TCAV). In Proc. ICML, Cited by: TABLE XVIII, §I-C, 2nd item. [54] P. Kindermans, S. Hooker, J. Adebayo, M. Alber, K. T. SchĂźt, S. Dähne, D. Erhan, and B. Kim (2019) The (un)reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, p. 267â280. External Links: Document Cited by: TABLE XVIII, item 2, §I, §X. [55] S. Krishna, T. Han, A. Gu, S. Wu, S. Jabbari, and H. Lakkaraju (2024) The disagreement problem in explainable machine learning: a practitionerâs perspective. Transactions on Machine Learning Research. Cited by: TABLE XVIII, §I, §X-G. [56] I. E. Kumar, C. Scheidegger, S. Venkatasubramanian, and S. A. Friedler (2020) Problems with Shapley-value-based explanations as feature importance measures. In Proc. ICML, Cited by: item 2, item 2, §X. [57] H. Lakkaraju, S. H. Bach, and J. Leskovec (2016) Interpretable decision sets: a joint framework for description and prediction. In Proc. ACM SIGKDD, Cited by: 4th item. [58] M. Li, H. Sun, Y. Huang, and H. Chen (2024) Shapley value: from cooperative game to explainable artificial intelligence. Autonomous Intelligent Systems 4 (1). Note: Art. no. 2 External Links: Document Cited by: item 1, §I-D, TABLE I. [59] P. Linardatos, V. Papastefanopoulos, and S. Kotsiantis (2021) Explainable AI: a review of machine learning interpretability methods. Entropy 23 (1), p. 18. Cited by: TABLE I, 4th item. [60] Z. C. Lipton (2018) The mythos of model interpretability. Communications of the ACM 61 (10), p. 36â43. Cited by: §I-B. [61] S. M. Lundberg, G. Erion, H. Chen, A. DeGrave, J. M. Prutkin, B. Nair, R. Katz, J. Himmelfarb, N. Bansal, and S. Lee (2020) From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence 2 (1), p. 56â67. Cited by: Appendix A, Appendix A, TABLE XIX, §XI-A, §IV-D, TABLE VI, TABLE IX, Axiom 7. [62] S. M. Lundberg and S. Lee (2017) A unified approach to interpreting model predictions. In Proc. NeurIPS, Vol. 30. Cited by: Appendix A, Appendix A, TABLE XVIII, TABLE XVIII, TABLE XIX, item 3, item 2, §I, §I-E, §I-B, §IV-C, §IV-E, §IV-E, TABLE VI, §VII-B, TABLE IX, TABLE IX, Axiom 7, Result 2, Result 2, Result 8. [63] D. Lundstrom and M. Razaviyayn (2025) Four axiomatic characterizations of the integrated gradients attribution method. Journal of Machine Learning Research 26 (177), p. 1â31. Cited by: TABLE XVIII, TABLE XIX, item 1, §I-E, 2nd item, §V-A, §V-F, Result 3, Result 3. [64] L. Merrick and A. Taly (2020) The explanation game: explaining machine learning models using Shapley values. In Proc. CD-MAKE, Cited by: §I-C. [65] C. Molnar (2022) Interpretable machine learning. 2nd edition, Independently published. External Links: Link Cited by: §I-D, TABLE I. [66] G. Montavon, S. Lapuschkin, A. Binder, W. Samek, and K. MĂźller (2017) Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognition 65, p. 211â222. Cited by: §VI-E. [67] M. B. Muhammad and M. Yeasin (2020) Eigen-CAM: class activation map using principal components. arXiv:2008.00299. Cited by: §VI-G. [68] W. J. Murdoch, P. J. Liu, and B. Yu (2018) Beyond word importance: contextual decomposition to extract interactions from LSTMs. In Proc. ICLR, Cited by: TABLE XIX, §XI-C. [69] M. Nauta, J. Trienes, S. Pathak, E. Nguyen, M. Peters, Y. Schmitt, J. SchlĂśtterer, M. van Keulen, and C. Seifert (2023) From anecdotal evidence to quantitative evaluation methods: a systematic review on evaluating explainable AI. ACM Computing Surveys 55 (13s). Note: Art. no. 295 External Links: Document Cited by: §I-D, TABLE I. [70] G. Owen (1972) Multilinear extensions of games. Management Science 18 (5), p. P64âP79. Cited by: §I-C. [71] V. Petsiuk, A. Das, and K. Saenko (2018) RISE: randomized input sampling for explanation of black-box models. In Proc. BMVC, Cited by: Appendix A, TABLE XVIII, TABLE XVIII, TABLE XVIII, TABLE XIX, §XII-C, §VII-E, §VII-I, TABLE IX, Figure 6, §IX-E, TABLE XIII. [72] G. Plumb, D. Molitor, and A. S. Talwalkar (2018) Model agnostic supervised local explanations. In Proc. NeurIPS, Cited by: TABLE XIX, §VII-B. [73] M. T. Ribeiro, S. Singh, and C. Guestrin (2016) âWhy should I trust you?â Explaining the predictions of any classifier. In Proc. ACM SIGKDD, p. 1135â1144. External Links: Document Cited by: Appendix A, TABLE XVIII, TABLE XIX, §I, §VII-B, TABLE IX. [74] M. T. Ribeiro, S. Singh, and C. Guestrin (2018) Anchors: high-precision model-agnostic explanations. In Proc. AAAI, Cited by: Appendix A, TABLE XIX, §VII-C, TABLE IX. [75] L. Rieger, C. Singh, W. J. Murdoch, and B. Yu (2020) Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. In Proc. ICML, Cited by: TABLE XVIII, 4th item, §IX-G. [76] A. S. Ross, M. C. Hughes, and F. Doshi-Velez (2017) Right for the right reasons: training differentiable models by constraining their explanations. In Proc. IJCAI, Cited by: TABLE XVIII, 4th item, §IX-G. [77] R. Saleem, B. Yuan, F. Kurugollu, A. Anjum, and L. Liu (2022) Explaining deep neural networks: a survey on the global interpretation methods. Neurocomputing 513, p. 165â180. Cited by: §I-D, TABLE I, 4th item. [78] W. Samek, A. Binder, G. Montavon, S. Lapuschkin, and K. MĂźller (2017) Evaluating the visualization of what a deep neural network has learned. IEEE Trans. Neural Netw. Learn. Syst. 28 (11), p. 2660â2673. Cited by: item Perturbation response.. [79] K. Schulz, L. Sixt, F. Tombari, and T. Landgraf (2020) Restricting the flow: information bottlenecks for attribution. In Proc. ICLR, Cited by: TABLE XIX, §X-D, §XIV-B, §VII-H. [80] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra (2017) Grad-CAM: visual explanations from deep networks via gradient-based localization. In Proc. ICCV, p. 618â626. Cited by: Appendix A, TABLE XVIII, TABLE XIX, item 2, §I, §VI-G, TABLE VIII, TABLE IX, Result 10. [81] S. Serrano and N. A. Smith (2019) Is attention interpretable?. In Proc. ACL, Cited by: TABLE XIX, §XI-C, §VI-H. [82] L. S. Shapley (1953) A value for n-person games. In Contributions to the Theory of Games, Volume I, H. W. Kuhn and A. W. Tucker (Eds.), Annals of Mathematics Studies, p. 307â317. Cited by: Appendix A, TABLE XVIII, TABLE XVIII, TABLE XIX, §I-A, §I, §IV, TABLE IX, Result 1. [83] A. Shrikumar, P. Greenside, and A. Kundaje (2017) Learning important features through propagating activation differences. In Proc. ICML, Cited by: Appendix A, TABLE XVIII, TABLE XIX, §I, §I-E, §V-C, §VI-A, §VI-D, TABLE VIII, TABLE IX, Axiom 15. [84] K. Simonyan, A. Vedaldi, and A. Zisserman (2013) Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv:1312.6034. Cited by: Appendix A, TABLE XVIII, TABLE XIX, §VI-A, TABLE VIII, TABLE IX. [85] C. Singh, W. J. Murdoch, and B. Yu (2019) Hierarchical interpretations for neural network predictions. In Proc. ICLR, Cited by: TABLE XVIII, §XI-C, 2nd item. [86] D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju (2020) Fooling LIME and SHAP: adversarial attacks on post-hoc explanation methods. In Proc. AAAI/ACM Conf. on AI, Ethics, and Society (AIES), p. 180â186. External Links: Document Cited by: TABLE XVIII, item 2, §I, §X-F, §X. [87] D. Smilkov, N. Thorat, B. Kim, F. ViĂŠgas, and M. Wattenberg (2017) SmoothGrad: removing noise by adding noise. arXiv:1706.03825. Cited by: Appendix A, TABLE XVIII, §VI-B, TABLE VIII, TABLE IX. [88] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller (2015) Striving for simplicity: the all convolutional net. In ICLR Workshop, Cited by: Appendix A, TABLE XIX, §VI-C, TABLE VIII, TABLE IX. [89] S. Srinivas and F. Fleuret (2019) Full-gradient representation for neural network visualization. In Proc. NeurIPS, Cited by: Appendix A, TABLE XIX, §VI-F, TABLE VIII, TABLE IX. [90] E. Ĺ trumbelj and I. Kononenko (2010) An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research 11, p. 1â18. Cited by: TABLE XIX, §IV-B. [91] P. Sturmfels, S. Lundberg, and S. Lee (2020) Visualizing the impact of feature attribution baselines. Distill. External Links: Document Cited by: TABLE XVIII, §X-B, 1st item, §V-B. [92] M. Sundararajan and A. Najmi (2020) The many Shapley values for model explanation. In Proc. ICML, Cited by: item 2, §XI-A, §I-C, §IV-D, §IV-H. [93] M. Sundararajan, A. Taly, and Q. Yan (2017) Axiomatic attribution for deep networks. In Proc. ICML, Proceedings of Machine Learning Research, Vol. 70, p. 3319â3328. Cited by: Appendix A, TABLE XVIII, TABLE XVIII, TABLE XVIII, TABLE XIX, item 1, §I, §I-D, §I, §V-F, TABLE VII, §V, TABLE IX, Result 3, Result 4. [94] R. Tomsett, D. Harborne, S. Chakraborty, P. Gurram, and A. Preece (2020) Sanity checks for saliency metrics. In Proc. AAAI, Vol. 34, p. 6021â6029. External Links: Document Cited by: TABLE XVIII, §I, §VIII-B, §IX-I. [95] M. Tsang, D. Cheng, and Y. Liu (2018) Detecting statistical interactions from neural network weights. In Proc. ICLR, Cited by: §IV-G. [96] M. Tsang, S. Rambhatla, and Y. Liu (2020) How does this interaction affect me? Interpretable attribution for feature interactions. In Proc. NeurIPS, Cited by: §XI-C. [97] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, P. Mardziel, and X. Hu (2020) Score-CAM: score-weighted visual explanations for convolutional neural networks. In Proc. CVPR Workshops, Cited by: Appendix A, TABLE XIX, §VI-G, TABLE VIII, TABLE IX. [98] Y. Wang, T. Zhang, X. Guo, and Z. Shen (2024) Gradient based feature attribution in explainable AI: a technical review. arXiv:2403.10415. Cited by: item 1, §I-D, TABLE I. [99] S. Wiegreffe and Y. Pinter (2019) Attention is not not explanation. In Proc. EMNLP, Cited by: §XI-C, §VI-H. [100] S. Xu, S. Venugopalan, and M. Sundararajan (2020) Attribution in scale and space. In Proc. CVPR, Cited by: Appendix A, TABLE XIX, §I-D, §V-C, TABLE VII, TABLE IX. [101] C. Yeh, C. Hsieh, A. Suggala, D. I. Inouye, and P. Ravikumar (2019) On the (in)fidelity and sensitivity of explanations. In Proc. NeurIPS, Cited by: TABLE XVIII, §IX-B, TABLE XIII, TABLE XIII. [102] M. D. Zeiler and R. Fergus (2014) Visualizing and understanding convolutional networks. In Proc. ECCV, p. 818â833. Cited by: Appendix A, Appendix A, §VI-C, TABLE VIII, §VII-A, TABLE IX, TABLE IX. [103] Q. Zheng, Z. Wang, J. Zhou, and J. Lu (2022) Shap-CAM: visual explanations for convolutional neural networks based on Shapley value. In Proc. ECCV, Cited by: TABLE XIX, §VI-G. [104] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba (2016) Learning deep features for discriminative localization. In Proc. CVPR, Cited by: TABLE XIX, §VI-G, Result 10. [105] L. M. Zintgraf, T. S. Cohen, T. Adel, and M. Welling (2017) Visualizing deep neural network decisions: prediction difference analysis. In Proc. ICLR, Cited by: §VII-A.