Paper deep dive
Defining Explainable AI for Requirements Analysis
Raymond Sheh, Isaac Monteath
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/20/2026, 8:42:37 PM
Summary
The paper proposes a unified framework for categorizing Explainable AI (XAI) requirements using three dimensions: Source (Post-Hoc Rationalisation vs. Introspective), Depth (Attribute vs. Model), and Scope (Justification vs. Teaching). It argues that existing literature often overlooks the distinction between explaining specific decisions (Justification) and general model behavior (Teaching), or the source of the explanation (black-box approximation vs. internal model logic). The authors illustrate these dimensions with examples from user experience, forensic compliance, neural networks, and decision trees to demonstrate how different applications require different levels of explainability.
Entities (15)
Relation Signals (13)
Isaac Monteath â authored â Defining Explainable AI for Requirements Analysis
confidence 99% · Defining Explainable AI for Requirements Analysis Raymond Sheh· Isaac Monteath
Raymond Sheh â authored â Defining Explainable AI for Requirements Analysis
confidence 99% · Defining Explainable AI for Requirements Analysis Raymond Sheh· Isaac Monteath
Explainable AI â categorizedby â Scope
confidence 95% · We present three dimensions for categorising the explanatory requirements of different applications. These are Source, Depth and Scope.
Explainable AI â categorizedby â Source
confidence 95% · We present three dimensions for categorising the explanatory requirements of different applications. These are Source, Depth and Scope.
Explainable AI â categorizedby â depth
confidence 95% · We present three dimensions for categorising the explanatory requirements of different applications. These are Source, Depth and Scope.
Source â hasvalue â Post-Hoc Rationalisation
confidence 92% · If it comes from another system or module... then we consider the source to be Post-Hoc Rationalisation.
Source â hasvalue â Introspective
confidence 92% · If instead the explanatory information came from the same process that produced the underlying decision... then we consider the source to be Introspective.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Explainable Artificial Intelligence (XAI) has become popular in the last few years. The Artificial Intelligence (AI) community in general, and the Machine Learning (ML) community in particular, is coming to the realisation that in many applications, for AI to be trusted, it must not only demonstrate good performance in its decisionmaking, but it also must explain these decisions and convince us that it is making the decisions for the right reasons. However, different applications have different requirements on the information required of the underlying AI system in order to convince us that it is worthy of our trust. How do we define these requirements? In this paper, we present three dimensions for categorising the explanatory requirements of different applications. These are Source, Depth and Scope. We focus on the problem of matching up the explanatory requirements of different applications with the capabilities of underlying ML techniques to provide them. We deliberately avoid including aspects of explanation that are already well-covered by the existing literature and we focus our discussion on ML although the principles apply to AI more broadly.
Tags
Links
- Source: https://arxiv.org/abs/2602.19071v1
- Canonical: https://arxiv.org/abs/2602.19071v1
Trouble viewing inline? Open PDF directly â
Full Text
32,524 characters extracted from source content.
Expand or collapse full text
Originally published as Sheh, R., Monteath, I. Defining Explainable AI for Requirements Analysis. K Ìunstl Intell 32, 261-266 (2018). https://doi.org/10.1007/s13218-018-0559-3 Defining Explainable AI for Requirements Analysis Raymond Sheh· Isaac Monteath Abstract Explainable Artificial Intelligence (XAI) has become popular in the last few years. The Artificial Intelligence (AI) community in general, and the Ma- chine Learning (ML) community in particular, is com- ing to the realisation that in many applications, for AI to be trusted, it must not only demonstrate good per- formance in its decisionmaking, but it also must ex- plain these decisions and convince us that it is making the decisions for the right reasons. However, different applications have different requirements on the infor- mation required of the underlying AI system in order to convince us that it is worthy of our trust. How do we define these requirements? In this paper, we present three dimensions for cate- gorising the explanatory requirements of different appli- cations. These are Source, Depth and Scope. We focus on the problem of matching up the explanatory require- ments of different applications with the capabilities of underlying ML techniques to provide them. We delib- erately avoid including aspects of explanation that are already well-covered by the existing literature and we focus our discussion on ML although the principles ap- ply to AI more broadly. Keywords Explainable AI· Machine Learning· Decision Trees Raymond Sheh and Isaac Monteath (formerly) Department of Computing, Curtin University, Bentley WA 6102, Australia E-mail: ray@raymondsheh.org 1 Introduction Artificial Intelligence (AI) in general, and Machine Learn- ing (ML) in particular, have achieved impressive levels of performance in replicating and exceeding human ca- pabilities in several high profile decision-making and behaviour generation tasks. Techniques such as Neu- ral Networks and Deep Learning [24] have always gen- erated remarkable results, ranging from work on self- driving cars [20] to beating human champions in games such as Go [27]. However, much of this advancement has come at the expense of the ability for humans to trace, understand, verify and learn from these intelli- gent systems. This is particularly important in applica- tions where systems must be traceable, reliable or held accountable for safety or regulatory purposes. Indeed, while good software engineering practice has resulted in software becoming more modular, reliable, testable, predictable and accountable, machine learned systems are becoming more tightly coupled and their interac- tions increasingly obfuscated and opaque [25]. This increasing obfuscation of the decisionmaking processes of AI agents and systems will inevitably lead to an erosion of trust. The increasing number of inci- dents in the popular media of AI systems making in- correct decisions has only accelerated this erosion. To be trusted, a system has to demonstrate competence (the capability to perform the task, make the decision or provide the information), honesty (that the process leading to the decision is transparent and accountable) and alignment (that the agent or system doesnât have an ulterior motive and hasnât been compromised) [10]. arXiv:2602.19071v1 [cs.AI] 22 Feb 2026 2Raymond Sheh, Isaac Monteath Explanations form a vital part of satisfying these re- quirements. Explainable AI (XAI) seeks to produce AI agents and systems that can not only make decisions but can also appropriately satisfy the applicationâs requirements for associated explanations. Although it can trace its origins to the earliest days of AI research, particularly in the Expert Systems community [29, 17], XAI has be- come a very topical area in the last couple of years. To properly develop and deploy XAI capabilities to answer these requirements, we need semantics that en- capsulate these various capabilities and requirements under one unified set of categories. This paper presents a survey of existing categorisations and highlights where they are insufficient. We then present a novel, unified set of categories that we propose will answer this need. The contribution of our work is a set of categories that enable those who specify, use, develop or otherwise rely on XAI systems to trade off the human need for ex- plainability against other factors such as efficiency, pre- dictive accuracy and representational flexibility of the underlying ML systems. Each category also outlines the needs of the aforementioned users. These categories are an evolution of those presented by [26] and complement existing human factors and cognitive science work on what explanations are necessary and how they should be presented. In a sense, the underlying concepts are not new and exist implicitly and in parts within other classifications. Our contribution is in bringing them to- gether into an explicit, unified set of semantics that can be used for requirements and capability analysis for the purpose of trust in AI systems. An infinite number of explanations can explain a given decision, just as an infinite number of functions can explain a single datapoint. This is true even ignor- ing factors of granularity, presentation and abstraction. Similarly, a subset of these explanations (although still possibly infinite) will explain any finite set of decisions on actions that the system or agent may make and only one explanation will truly match the systemâs or agentâs underlying decision process. Much of the existing commentary on explainability, especially from the cognitive science perspective, is ac- cepting of the idea that an explanation can be useful if it matches the decisions, without being some version of the âone trueâ explanation. For example, [3] assert that understandability is the most important problem, even beyond the source of the explanation (which they re- fer to as âaccuracyâ). This may be true when we have no way of knowing what the âone trueâ explanation is. This is often the case for outward-looking explana- tions. We may never truly know why a company went bankrupt although we may have some good theories. Crucially, this is also often the case for human expla- nations where so much of the decision-making process is not the result of conscious thought. 2 Background There is much existing work on defining and categoris- ing explanations, especially from the cognitive science perspective. Some, such as [16], are concerned with how to focus explanations onto what is most informative or desirable for a given audience and how this changes with application. Other work such as [31] focus on how AI systems can use explanations in dialog to convincingly justify their decisions. [18] discuss why it is important to âbeware of the inmates running the asylumâ, not- ing that explanations should be targeted towards the right user audience â who are not necessarily the AI practitioner, software engineer, or even the domain ex- pert. This may be important from the perspective of presentation to the end user. However, we argue that it is precisely the AI practitioners, software engineers or domain experts who place greater demands on where the underlying information for explanations come from. Therefore, it is important to consider this as a factor in the discussion, separate from the presentation of the explanation itself. With so many groups working on the topic of ex- plainability, even the semantics surrounding explain- ability, such as its definition, requirements and metrics, have become overloaded. Recent work brings together some of these semantics but also highlights where they are deficient, particularly with regard to the source and detail available in the underlying information. This makes these properties difficult to evaluate in an intelligent system or agent. [3] also make an important distinc- tion between the terms âinterpretable modelâ, which requires an underlying âwhite-boxâ model such as a decision tree, and âinterpretable representationâ which refers to how any underlying model is represented to the end-user. Although their paper highlights many of the varied applications where the requirements are poorly covered by existing semantics, they do not propose a Defining Explainable AI for Requirements Analysis3 unified set of categories to discuss these at a level of fidelity necessary to properly consider the trade-offs. [19] defines explanations as distinct from interpre- tations. An explanation is â... the collection of features of the interpretable domain, that have contributed for a given example to produce a decision ...â while an in- terpretation is â... the mapping of an abstract concept (e.g. a predicted class) into a domain that the human can make sense of.â We feel that this definition of expla- nation is unnecessarily narrow. There are many times when we demand more of an explanation than simply the collection of features. We want to know more about what the system knows in its decision-making. According to [8], current techniques for evaluating interpretability fall into two categories. The first pro- poses that if an AI agent or system is useful in some practical application (or a simplified version of it), then there must be a way to interpret the logic behind its ac- tions. The second involves using a model that is intrin- sically interpretable, and fitting it to solve a problem required by the application. While the interpretability of both approaches can be tested using human-beings, neither provides a rigorous framework for quantifying interpretability. 3 Categorising Explanations As introduced earlier, we divide explanations along three dimensions: Source, Depth and Scope. These are vi- sualised in Figure 1 along with the placement of some examples of different AI techniques. In this section we present our definitions for these dimensions. We consider the Source of the explanation to be where the explanatory information came from. If it comes from another system or module, or from a system ob- serving a decision-making process as a black box and generating its explanations with no more information than another observer might have, then we consider the source to be Post-Hoc Rationalisation. If instead the ex- planatory information came from the same process that produced the underlying decision and, crucially, retains symbolic meaning from that underlying decision (as op- posed to purely statistical information, for example), then we consider the source to be Introspective. While much of the discussion around XAI would seem to tend towards one end of this category or the other, as we will discuss, there is also a continuum between these two extremes. Fig. 1 Three dimensions of explanation with examples of the explanatory capabilities possible using different techniques. Axes are arbitrary. Note that each technique covers a âcubeâ that extends to the lower left corner but has been omitted for clarity. The distinction between Post-Hoc Rationalisation and Introspective explanations appears in most cate- gorisations of explanation. For example, [30] refer to this as âFidelityâ and [3] as âAccuracyâ, while the âOpaqueâ category [7] corresponds to Post-Hoc Rationalisation in our vernacular. The purely Introspective explanation corresponds to the âone trueâ explanation while oth- ers are Post-Hoc Rationalisation. To the best of our knowledge, most XAI systems in the contemporary ML literature tend to be Post- Hoc Rationalisation systems of varying degrees. This is perhaps due to the interest in Deep Learning, which is necessarily opaque [21, 22, 15]. Post-Hoc Rationalisation Explanations may only be valid around a local decision or some (perhaps often-visited or optimised) subset of the state space, or else they may attempt to be globally valid. Just as in function fitting, the nature of the devi- ation will depend on how the explanation is generated. In the general case, there is no way to know the location or extent of these deviations beyond probing the space, thereby limiting the ability of Post-Hoc Rationalisation Explanations to predict the systemâs decisions beyond instances seen in operation or training. Introspective Explanations are based on the systemâs underlying decision process. As such, they are the ac- curate, âtrueâ explanation for a given decision, to a particular level of abstraction. Such AI techniques and models are often referred to as being âexplainableâ or âtransparentâ. 4Raymond Sheh, Isaac Monteath It is vital to know if an explanation is Post-Hoc Ra- tionalisation or Introspective when used for correcting faults, predicting behaviour in critical systems, or for compliance and accountability. Post-Hoc Rationalisa- tion explanations may explain the observed behaviour but have limited relevance to the underlying decision or fault. However, it may be impossible for the end-user to tell the difference between these types of explanation. In dividing explanations and techniques between Post- Hoc Rationalisation and Introspective, it is important to determine what is meaningful in a given context. Ev- ery program that can run on a deterministic computer is arguably capable of an âIntrospectiveâ explanation. In the limiting case, all one needs to do is to dump out the memory contents at every instruction. Moving up one level of abstraction, a conventional deep neural network can also be considered âIntrospectiveâ insofar as it is possible to simply print out all of the neural network weights and activations. For all but the most trivial problems, these âExecutionâ versions of intro- spective explanations are neither understandable nor efficient enough to be useful. The Depth of an explanation can be Attribute or Model. An explanation that describes how attributes were used to make decisions within the context of the model are referred to as Attribute explanations. Those that also include how the model was generated from training data and background knowledge are referred to as Model explanations. For example, an image clas- sification system may provide an Attribute explanation in the form of a saliency map. If it were then able to answer a question about how it arrived at that saliency map, such as from training data or background knowl- edge, it would be considered capable of providing Model explanations. We find that Model explanations are largely forgot- ten in the XAI literature. The vast majority of tech- niques, discussions and semantics assume that expla- nations are limited to explaining how an AI system makes use of the attributes being observed to gener- ate its current answer. This distinction is important because it speaks to the ability of a system to assist humans in uncovering the reason for failures, both for troubleshooting and accountability purposes. After all, if an image recognition system on an autonomous mil- itary agent incorrectly recognises a turtle as a weapon [1], then not only would the humans in charge be in- terested in knowing the Attribute explanation but they would also want the Model explanation, which may sug- gest where else the agent will make the same mistake and, ideally, how to fix or mitigate the problem. Within Attribute explanations we also consider a sub-dimension, which we define as Attribute Identity vs Attribute Use explanations. A system that is only capa- ble of Attribute Identity explanations is able to describe which attributes were used in making its decision but is unable to explain, in a way that is meaningful to humans, how they were used. The definition of expla- nation proposed in [19] only covers Attribute Identity explanations. Examples of these explanations include statistics on which attributes were most important and saliency maps. Attribute Use explanations are able to explain how those attributes were used. A decision tree or logic rule chain are examples of Attribute Use explanations. It is important to note that meaningfulness can be nebu- lous. In the general case we consider an explanation to be Attribute Use if the AI system is able to incorporate information about decision boundaries into its explana- tion, be it Introspectively (by looking at the underlying model) or in a Post-Hoc Rationalisation fashion (such as by sampling). Explanation Scope can be Justification or Teach- ing. If the intent of the explanation is focused on a spe- cific decision or group of decisions, it is a Justification Explanation. Otherwise, it is a Teaching Explanation. Teaching Explanations tend to be forgotten in the XAI literature. The work of [4] comes closest in acknowledg- ing that there are explanations for justifications. Like the aforementioned Model explanations, we find Teaching explanations to be largely ignored by the ex- isting literature on explainability. However, they are an important part of forming trust with an AI system, par- ticularly in terms of its ability to properly generalise outside of examples that have been seen. These expla- nations can take many forms. At one end they can still be rooted in instances. For example, answers to the in- quiry âWhat are examples of other instances that you consider to be similar to this?â would be a Teaching explanation. These other instances displayed may be from the past (training or otherwise) or they may be synthetic instances, generated by the model. Beyond these explanations are those that concern understanding the decision boundaries. âHow far would I need to move this attribute to change your decision?â might be such a question that would be asked of a high Defining Explainable AI for Requirements Analysis5 reliability system. The complexity of the resulting an- swer could be considerable, especially for models or ap- plications where decision boundaries are complex. At the other end are more abstract concepts. âWhat are at- tributes that were significant in deciding on this class?â or even, âTeach me about the differences between these classesâ, for instance. 4 Discussion In this section, we will discuss some salient examples, first of application requirements and then of AI tech- nique capabilities. The intent is to highlight some issues in using these categorisations, rather than to be com- prehensive. 4.1 Application: User Experience An AI system, such as a service robot, might be asked why they made a decision or exhibited a behaviour that was surprising. The human has a model for the robotâs behaviour that is different from the robotâs observed behaviour. The role of explanation is to help the user to better understand the agentâs behaviour, to be more comfortable and to make better use of it by performing âmodel reconciliationâ [6]. This goes both ways. For a good user experience, sometimes the robot should change its behaviour to make it fit the humansâ model, to make it more âexplicableâ. In our view, such an application does not require ex- planation to be Introspective. Indeed, it may be better to select an explanation that the user is more likely to understand or accept over what is true, so long as it ex- plains the robotâs behaviour with sufficient plausibility where it is likely to be observed. In terms of explana- tion depth, we would suggest that such explanations would be mainly Attribute Identity or Use. The excep- tion may be in situations where the agent or system is expected to learn on-the-fly. A service robot being asked âWhy did you go the long way around the building?â and answering âBecause Fred told me to go that way last week.â is providing an explanation at the Model Depth. The combination of a Model Depth with Post- Hoc Rationalisation Source might seem contradictory. We consider such an explanation as explaining the de- cision in a Post-Hoc Rationalisation manner given the whole corpus of data available to the system â training, past experience and current instance. In contrast, an explanation at the Attribute Depth only considers the current instance and model. 4.2 Application: Forensic and Compliance Forensic and Compliance applications range from pub- lic safety (such as self-driving cars) to privacy and ethics, such as the European Unionâs General Data Protection Regulation (GDPR) [12]. In our categorisation, the ex- planations demanded in these applications would have a Source of Introspective and a Scope of Model. The human users demand the true reason for an AI systemâs decisions (Depth of Justification) or some understand- ing as to what the AI system will do more generally (Depth of Teaching ). In such an application, the use of AI techniques that do not readily provide these ca- pabilities should be used with extreme caution as ver- ification of their performance is going to be at least in part dependent on statistical behaviour. Interestingly, the highest demands placed on explanation in terms of information source by this application could perhaps also be the least demanding from a human factors per- spective. The human users here are heavily invested in understanding the AI systemâs decisions. 4.3 AI: Neural Networks There has been much recent discussion about explain- ability with regards to Neural Networks and associated Deep Learning techniques. Some say they are not ex- plainable [9, 23] while others claim that it is quite possi- ble to generate explanations from them [21, 14]. In our observation, much of this disagreement comes from a disagreement in the types of explanation that are ex- pected. Some sectors of the ML community do not con- sider any explanations beyond those that we would cat- egorise as Attribute Identity, Justification explanations â the definition by [19] as discussed in Section 2 being one example. By such a narrow definition, Neural Net- works can be made explainable. Having said that, [19] does survey work that is able to move towards more Introspective and Attribute Use explanations. In this direction, there has been much recent inter- est in tricking deep learning based image recognition system into reporting arbitrary results through minor perturbations in the image [28, 15] or placing a separate 6Raymond Sheh, Isaac Monteath object in the scene [5]. Approaches that seek to explain such systems by highlighting saliency in the images [2] are helpful to at least detect and identify the problem. More generally, other techniques such as [21] attempt to analyse the behaviour of the network in a black-box fashion for the purpose of increasing trust in the system. In many applications, such as finding similar images or products, especially where there is no adversary, this is all one might expect. Some performers on the DARPA XAI program [13] are generating explanations from deep learning by lever- aging information from within the network. This may be by inspecting the network after it has been trained, such as by using input gradients [22], or by incorpo- rating the learning of explanations into the learning of the network itself [14]. Through this, significant in- termediate representations and decision boundaries can be found. Some of these explanations are partially In- trospective, Attribute Use, Justification explanations. These explanations are still not necessarily truly âIn- trospectiveâ insofar as thereâs no guarantee that they are the result of the exact same network operation that generated the decision. However, it at least opens up the possibility that they may use the same interme- diate representations, providing a partial grounding in the decision-making process. 4.4 AI: Decision Trees and Rule Learners Decision Trees, Inductive Logic Programming, Ripple- Down Rules and other representations that lend them- selves to rule-based representations tend to be thought of as explainable and transparent. They can yield expla- nations that satisfy all eight octants of our classification scheme, including the most âdifficultâ, being Introspec- tive, Model, Teaching explanations. Decision trees can be instrumented in order to ob- tain some of this underlying information. For example, metadata for Model explanations can be retained from the training process or inferred after the fact. Knowl- edge of the information theoretic underpinnings of how the tree is learned also furnish information that can be used to show which rules were most important, which were best supported by the current instance, and which other rules from other branches might have been rele- vant had the instance changed slightly. One must be careful in claiming that such models are usefully explainable just because they have a trace- able, perhaps deterministic path from training data to model to decision. If the underlying phenomena be- ing predicted cannot be meaningfully expressed by the model then any explanation will be forced and limited in its usefulness, even if the predictive accuracy is suffi- cient. For example, a decision tree can learn a concept in 2D space with a decision boundary defined byy =x but it does so by âshatteringâ this simple concept into a multitude of If-Then-Else rules that are as informa- tive to a human as the weights in a Neural Network. In our categorisation, such a decision tree does not sup- port Introspective or Teaching explanations on these attributes. 5 Conclusion Before we can match an applicationâs explanatory re- quirements with AI capabilities, we need to define ex- planation. There have been many proposed definitions in the literature, some salient examples of which we have surveyed. However, we have found that they tend to be overly narrow and, generally, focused on the (equally important problem of) presentation of the explanation rather than the source. As AI becomes more a part of critical decision-making, society as a whole is be- ginning to demand more reliability, transparency and accountability from AI agents and systems. However, explainability is a trade-off, often occurring at the ex- pense of other factors such as performance or develop- ment effort. In this paper, we have presented a novel categorisation for explanations along three dimensions of Source, Depth and Scope, and discussed these in the context of several salient examples of applications and AI techniques. We propose that these cover the full space of explanations and can be used to match appli- cationsâ explanatory requirements with AI capabilities to satisfy them. Semantics for defining and categorising explanation is just part of the challenge in matching explanatory requirements with AI capabilities. Another part of the challenge is extending these semantics to define how ârepairableâ an AI system is based on its underlying AI techniques. In some cases it may well be acceptable to simply retrain on the misclassified examples. However, in other cases it may be necessary to have verifiable proof that not only has a particular instance of error been solved, but that all similar potential errors have also been solved without âcatastrophically forgettingâ Defining Explainable AI for Requirements Analysis7 [11] what the AI system might have otherwise done cor- rectly. Certainly mechanical infrastructure â vehicles, for instance â are held to this standard, it is only a matter of time before certain AI systems are held to similar standards. Augmenting our categories with the semantics to describe these orthogonal requirements is the subject of future work. Finally, metrics for expla- nation apply differently to the different categories of explanation that we presented. Developing these met- rics is also the subject of future work. References 1. Athalye, A., Engstrom, L., Ilyas, A., Kwok, K.: Syn- thesizing Robust Adversarial Examples (2017). URL http://arxiv.org/abs/1707.07397 2. Bach, S., Binder, A., Montavon, G., Klauschen, F., M Ìuller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE 10(7), 1â46 (2015) 3. Bibal, A.: Interpretability of Machine Learning Models and Representations : an Introduction. Proc. ESANN (April), 77â82 (2016) 4. Biran, O., Cotton, C.: Explanation and Justification in Machine Learning: A Survey.IJCAI XAI Workshop (2017) 5. Brown, T.B., Man Ìe, D., Roy, A., Abadi, M., Gilmer,J.:AdversarialPatch(2017).URL http://arxiv.org/abs/1712.09665 6. Chakraborti,T.,Sreedharan,S.,Zhang,Y., Kambhampati,S.:PlanExplanationsasModel Reconciliation.Proc.IJCAI(2016).URL http://arxiv.org/abs/1701.08317v3 7. Doran, D., Schulz, S., Besold, T.R.: What Does Explain- able AI Really Mean? A New Conceptualization of Per- spectives (2017). URL http://arxiv.org/abs/1710.00794 8. Doshi-velez, F., Kim, B.: Towards A Rigorous Sci- ence of Interpretable Machine Learning (2017). URL http://arxiv.org/abs/1702.08608v2 9. Doyle, D., Tsymbal, A., Cunningham, P.: A review of ex- planation and explanation in case-based reasoning. Tech. rep., Trinity College Dublin, Department of Computer Science (2003) 10. Freed, M.: Three elements of trust. Private communica- tion (2018) 11. French, R.M.: Catastrophic forgetting in connectionist networks. Trends in cognitive sciences 3(4), 128â135 (1999) 12. Goodman, B., Flaxman, S.: European Union regulations on algorithmic decision-making and a âright to expla- nationâ. In: Proceedings of the 2016 ICML Workshop on Human Interpretability in Machine Learning (2016). URL https://arxiv.org/pdf/1606.08813 13. Gunning, D.: Explainable Artificial Intelligence (XAI). Broad Agency Announcement DARPA-BAA-16-53, De- fence Advanced Research Projects Agency (2016) 14. Hendricks, L.A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., Darrell, T.: Generating visual explanations. In: European Conference on Computer Vision, p. 3â19. Springer (2016) 15. Huang, X., Kwiatkowska, M., Wang, S., Wu, M.: Safety verification of deep neural networks. Lecture Notes in Comp. Sci. 10426 LNCS, 3â29 (2017) 16. Keil, F.C.: Folkscience: coarse interpretations of a com- plex reality. Trends in Cog. Sci. 7(8), 368â373 (2003) 17. van Lent, M., Laird, J.E.: Learning procedural knowl- edge through observation. In: Proc. 1st Intâl Conf. on Knowledge Capture, p. 179â186. ACM (2001) 18. Miller, T., Howe, P., Sonenberg, L.: Explainable AI : Be- ware of Inmates Running the Asylum. In: IJCAI Work- shop on XAI (2017) 19. Montavon, G., Samek, W., M Ìuller, K.R.: Methods for in- terpreting and understanding deep neural networks. Dig- ital Signal Processing: A Review Journal 73, 1â15 (2018) 20. Pomerleau, D.A.: Alvinn: An autonomous land vehicle in a neural network. In: Advances in neural information processing systems, p. 305â313 (1989) 21. Ribeiro, M.T., Singh, S., Guestrin, C.: Why should i trust you?: Explaining the predictions of any classifier. In: Pro- ceedings of the 22nd ACM SIGKDD international confer- ence on knowledge discovery and data mining, p. 1135â 1144. ACM (2016) 22. Ross, A.S., Hughes, M.C., Doshi-Velez, F.: Right for the right reasons: Training differentiable models by con- straining their explanations. In: Proc. IJCAI (2017). URL http://arxiv.org/abs/1703.03717 23. Roth-Berghofer, T.R.: Explanations and case-based rea- soning: Foundational issues. In: European Conference on Case-Based Reasoning, p. 389â403. Springer (2004) 24. Schmidhuber, J.: Deep learning in neural networks: An overview. Neural Networks 61, 85â117 (2015) 25. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M.: Machine Learning: The High Interest Credit Card of Technical Debt. In: NIPS Workshop on SE4ML (2014) 26. Sheh, R.: Different XAI For Different HRI. In: AAAI FSS Workshop on AI-HRI (2017) 27. Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484â489 (2016) 28. Su, J., Vargas, D.V., Sakurai, K.: One pixel attack for fooling deep neural networks. CoRR abs/1710.08864 (2017) 29. Swartout, W.R.: XPLAIN: a system for creating and ex- plaining expert consulting programs. Artificial Intelli- gence 21(3), 285â325 (1983) 30. Swartout,W.R.,Moore,J.D.:Explanationin Second Generation Expert Systems.In: Sec- ond Generation Expert Systems. Springer Berlin Heidelberg,Berlin,Heidelberg(1993).URL http://w.springerlink.com/index/10.1007/978-3- 642-77927-5 24 31. Tolchinsky, P., Modgil, S., Atkinson, K., McBurney, P., Cort Ìes, U.: Deliberation dialogues for reasoning about safety critical actions. Autonomous Agents and Multi- Agent Systems 25(2), 209â259 (2012)