Paper deep dive
Beyond Accuracy: An Explainability-Driven Analysis of Harmful Content Detection
Trishita Dhara, Siddhesh Sheth
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/20/2026, 1:24:14 PM
Summary
This paper presents an explainability-driven analysis of a RoBERTa-based harmful content detection model trained on the Civil Comments dataset. Using Shapley Additive Explanations (SHAP) and Integrated Gradients, the authors analyze correct predictions and systematic failures (false positives/negatives). The study finds that SHAP focuses on explicit lexical cues, leading to over-attribution in borderline cases, while Integrated Gradients provide diffuse contextual attributions. The work highlights that explainability serves as a diagnostic and transparency resource for human-in-the-loop moderation rather than a performance-enhancing lever.
Entities (7)
Relation Signals (7)
RoBERTa → trainedon → Civil Comments
confidence 98% · A RoBERTa-based classifier [1] is fine-tuned on the Civil Comments dataset
Shapley Additive Explanations → appliedto → RoBERTa
confidence 97% · Shapley Additive Explanations and Integrated Gradients are used to analyze the behavior of a RoBERTa-based classifier
Integrated Gradients → appliedto → RoBERTa
confidence 97% · Integrated Gradients are used to analyze the behavior of a RoBERTa-based classifier
Shapley Additive Explanations → exhibitsbehavior → focused_attributions_on_explicit_lexical_cues
confidence 95% · Shapley Additive Explanations extract more focused attributions on explicit lexical cues
Integrated Gradients → exhibitsbehavior → diffuse_contextual_attributions
confidence 95% · Integrated Gradients appear to extract more diffuse contextual attributions
Explainable AI → supports → Human-in-the-loop moderation
confidence 94% · explainable AI can foster human-in-the-loop moderation by exposing model uncertainty
Shapley Additive Explanations → causes → false_positives
confidence 92% · The consequent divergence in their outputs manifests in both false negatives and false positives... Shapley Additive Explanations... misclassify the example as toxic token by token
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Although automated harmful content detection systems are frequently used to monitor online platforms, moderators and end users frequently cannot understand the logic underlying their predictions. While recent studies have focused on increasing classification accuracy, little focus has been placed on comprehending why neural models identify content as harmful, especially when it comes to borderline, contextual, and politically sensitive situations. In this work, a neural harmful content detection model trained on the Civil Comments dataset is analyzed explainability-drivenly. Two popular post-hoc explanation methods, Shapley Additive Explanations and Integrated Gradients, are used to analyze the behavior of a RoBERTa-based classifier in both correct predictions and systematic failure cases. Despite strong overall performance, with an area under the curve of 0.93 and an accuracy of 0.94, the analysis reveals limitations that are not observable from aggregate evaluation metrics alone. Integrated Gradients appear to extract more diffuse contextual attributions while Shapley Additive Explanations extract more focused attributions on explicit lexical cues. The consequent divergence in their outputs manifests in both false negatives and false positives. Qualitative case studies reveal recurring failure modes such as indirect toxicity, lexical over-attribution, or political discourse. The results suggest that explainable AI can foster human-in-the-loop moderation by exposing model uncertainty and increasing the interpretable rationale behind automated decisions. Most importantly, this work highlights the role of explainability as a transparency and diagnostic resource for online harmful content detection systems rather than as a performance-enhancing lever.
Tags
Links
- Source: https://arxiv.org/abs/2603.18015v1
- Canonical: https://arxiv.org/abs/2603.18015v1
Trouble viewing inline? Open PDF directly →
Full Text
27,963 characters extracted from source content.
Expand or collapse full text
Beyond Accuracy: An Explainability-Driven Analysis of Harmful Content Detection Trishita Dhara 1[0000−0002−9325−9286] and Siddhesh Sheth 2[0009−0003−7154−0477] 1 Upper Hand, Indianapolis, IN 46204, USA trishitadhara123@gmail.com 2 Ace Rent a Car, Indianapolis, IN 46204, USA shethsiddhesh268@gmail.com Abstract. Although automated harmful content detection systems are frequently used to monitor online platforms, moderators and end users frequently cannot understand the logic underlying their predictions. While recent studies have focused on increasing classification accuracy, little focus has been placed on comprehending why neural models identify content as harmful, especially when it comes to borderline, contextual, and politically sensitive situations. In this work, a neural harmful con- tent detection model trained on the Civil Comments dataset is ana- lyzed explainability-drivenly. Two popular post-hoc explanation meth- ods, Shapley Additive Explanations and Integrated Gradients, are used to analyze the behavior of a RoBERTa-based classifier in both correct predictions and systematic failure cases. Despite strong overall perfor- mance, with an area under the curve of 0.93 and an accuracy of 0.94, the analysis reveals limitations that are not observable from aggregate evaluation metrics alone. Integrated Gradients appear to extract more diffuse contextual attributions while Shapley Additive Explanations ex- tract more focused attributions on explicit lexical cues. The consequent divergence in their outputs manifests in both false negatives and false positives. Qualitative case studies reveal recurring failure modes such as indirect toxicity, lexical over-attribution, or political discourse. The results suggest that explainable AI can foster human-in-the-loop mod- eration by exposing model uncertainty and increasing the interpretable rationale behind automated decisions. Most importantly, this work high- lights the role of explainability as a transparency and diagnostic re- source for online harmful content detection systems rather than as a performance-enhancing lever. Keywords: Explainable AI· Trustworthy AI· Natural Language Pro- cessing· Content Moderation· Harmful Content Detection· Model In- terpretability. 1 Introduction In order to detect and control harmful content, such as hate speech, harass- ment, and toxic language, online platforms are depending more and more on arXiv:2603.18015v1 [cs.CL] 24 Feb 2026 2T. Dhara and S. Sheth automated systems. Machine learning models have become essential to content moderation workflows as the amount of user-generated content keeps increasing because they allow for the quick and extensive identification of potentially haz- ardous content. The detection performance across established benchmarks has significantly improved due to recent developments in pretrained language models [1, 9]. Despite recent advances, harmful content detection remains a challenging problem with significant ethical and social implications. False negatives allow abusive or harmful behavior to persist, whereas false positives risk suppress- ing legitimate expression. As a result, accuracy based evaluation in itself fails to provide a complete view of system performance in real-world moderation settings. Human moderators must have access to interpretable rationales under- lying model decisions in order to assess accountability, consistency, and fairness of the system. In the absence of such transparency, the responsible deployment of content moderation systems becomes difficult. Explainable artificial intelligence (XAI) helps to show why and how a system made a certain decision. It has been a popular research area in recent years, to make machine learning models more trustworthy and transparent. Post-hoc XAI methods like Shapley Additive Explanations [3] and Integrated Gradients [4], ex- plain a model’s predictions by showing how each input feature influenced the re- sult without modifying the underlying model architecture. Although these tech- niques are increasingly adopted into decision-support tools, structured analysis of their behavior and reliability remains limited, particularly in the domain of harmful content moderation. Previous research in the domain of harmful content detection mostly focus on improving classification recall through new models, better pretraining strategies, or extending datasets [5, 2]. Even though explana- tion methods have been applied to text classification in general, relatively less work studies how the explanation patterns varies between correct and incorrect predictions in harmful content detection. The study addresses this gap by imple- menting an explanation-based analysis of a harmful content detection model. A RoBERTa-based classifier [1] is fine-tuned on the Civil Comments dataset [5] and then examined using Shapley Additive Explanations and Integrated Gradients to understand model predictions. The analysis focuses on interpreting the model outputs rather than proposing new architecture, particular attention is given to ambiguous and context-sensitive cases commonly encountered in moderation practice. Both quantitative and qualitative assessments were conducted to un- derstand failure modes of transformer based harmful content detection models. The rest of the paper is organized into the following sections - related works, methodology, results, discussion and conclusion, and limitations. 2 Related Work 2.1 Harmful Content and Toxic Language Detection A lot of research has been done on automated detection of toxic and harmful language as online platforms try to filter vast amounts of user-generated content. Explainability-Driven Analysis of Harmful Content Detection3 Early approaches relied on traditional machine learning models with manually constructed features, like lexical and syntactic cues [6, 7]. When deep learning was introduced, neural architectures such as convolutional and recurrent net- works performed better on toxicity and hate speech benchmarks [2, 8]. Pretrained transformer-based language models have become the most pop- ular method for detecting toxic content in more recent research. By accurately simulating contextual semantics, models such as BERT and RoBERTa produce impressive results on a variety of datasets [1, 9]. Large-scale benchmarks, like the Civil Comments dataset, have made it possible to systematically analyze toxicity classifiers’ bias, robustness, and generalization [5, 10]. However, it is still very difficult to correctly identify subtle and context-dependent forms of toxicity, such as political discourse and implicit harassment. 2.2 Explainable Artificial Intelligence for Text Classification In order to improve machine learning systems’ transparency and credibility, ex- plainable artificial intelligence has emerged as a key area of study. Interpreting model predictions without changing the underlying architectures is the goal of post-hoc explanation techniques. Earlier works in explainability have shown that these explanations can assist in exposing false correlations and latent artifacts of the data, but are also vulnerable to model sensitivity due to instabilities, input perturbations and uncertainty changes indicated by perplexity shifts [13]. The faithfulness and reliability of post-hoc methods used for explanations on deep neural models still remain debated and investigated. 2.3 Explainability in Harmful Content Moderation Explainability has been studied extensively with natural language processing research works in general but not specifically applied to harmful content mod- eration. Most of the works examine explanation methods as audit mechanisms that detect bias in toxicity classifiers that focus mainly on identity terms and demographic attributes [5, 10]. Some other works have introduced explanation- oriented moderation interfaces that help human moderators understand auto- mated decisions in content classification tasks [14]. Despite these works, most of literature has approached question of expla- nation methods independently or has mainly emphasized fairness-driven goals, instead of attempts to systematically analyze and characterize explanation be- havior on both successes and diagnostic failure cases. There seem to be little empirical studies from the comparative view of explanation methods that are related to false positives and false negatives in the harmful content identifica- tion task. This work builds on preceding studies by presenting a comparative study of Shapley Additive Explanations and Integrated Gradients with a neu- ral toxicity classifier that helps extract insights regarding the behavior of these explanation methods and their errors in content moderation task. 4T. Dhara and S. Sheth 3 Methodolody 3.1 Task Description A binary toxic content identification task is formed where each input text is ei- ther identified as a toxic or a non-toxic class. The binary labels are obtained from the continuous toxicity scores of Civil Comments dataset [5] . This binary label- ing is achieved from a fixed threshold of 0.5 such that text inputs identified with a toxicity score greater than or equal to 0.5 are considered as toxic. This thresh- olding is similar to numerous models in previous studies. This task definition fits the normal paradigm of real-world scenarios where automated moderation systems are required to perform binary classification on users’ contents 3.2 Dataset Civil Comments dataset [5] is public comments with its metadata from 50 news websites across the globe. This dataset has been used in previous works to assess the robustness of models detecting harmful content and to study bias in auto- matic moderation systems [5, 10]. In the experiments, a subset of the dataset is used with 20,000 training instances, 4,000 validation instances and 4,000 test in- stances to be able to train a baseline model but keeping the costs reasonably low and the explanation behavior examinable. Although the reported explainability patterns are derived from a subset of the dataset, they capture model behavior shaped by learned lexical and contextual representations. Hence, these patterns are expected to generalize to larger samples and to other similar toxicity bench- marks. Extending the analysis to full-scale datasets remains a potential direction for future work. 3.3 Model Architecture RoBERTa-base [1] is selected as the classification model for this study. The model is a transformer-based language model that builds upon the BERT archi- tecture [9] through optimized pretraining strategies, including dynamic mask- ing and training on substantially larger text corpora. Its strong empirical per- formance across a wide range of natural language processing tasks makes it a suitable baseline for explainability analysis without introducing additional ar- chitectural complexity. A linear classification head is attached to the final hid- den representation corresponding to the special classification token, and the full model is fine-tuned end-to-end for binary toxic content classification. 3.4 Pipeline Overview Figure 1 illustrates the analysis pipeline employed in this work. Raw text inputs are first processed by the RoBERTa-based classifier to produce binary toxicity predictions. Next, post-hoc explanation techniques, namely Shapley Additive Explanations and Integrated Gradients, are executed on the model after training Explainability-Driven Analysis of Harmful Content Detection5 to yield token-level attribution score for individual model predictions. The model explanations for true positives, false positives, and false negatives are analyzed to discover any patterns, and failure modes that are consistent and applicable to automated content moderation. Fig. 1. Overview of the harmful content detection and explainability analysis pipeline. This pipeline implements a practical moderation workflow where automated predictions are enhanced with explanations to facilitate human moderation and auditing. 3.5 Training Configuration Model fine-tuning is performed using the AdamW optimizer with a batch size of 32 and a learning rate of 2× 10 −5 . The model is trained for one epoch, which is enough to achieve good prediction quality and prevents overfitting. The performance of the model is evaluated at the end of the training epoch in the validation set. Harmful content evaluation metrics are defined on accuracy, precision, recall, F1-score, and area under the receiver operating characteristic curve for the toxic class on a held-out test set. These metrics are used in harmful content detection settings as they reflect the trade-offs between false positive and false negative errors while addressing class imbalance [5, 2]. 3.6 Explainability Methods To understand model behavior beyond just overall performance measures, two post-hoc explainability techniques are applied: Shapley Additive Explanations and Integrated Gradients. Shapley Additive Explanations. Shapley Additive Explanations are inspired by cooperative game theory and estimate the contribution of each input token to the prediction of a model [3]. As a model-agnostic approach, the method yields token- level attributions that facilitate direct interpretation of individual predictions. 6T. Dhara and S. Sheth Integrated Gradients. Integrated Gradients calculates token attributions by in- tegrating gradients along a continuous path from a baseline input to the final input [4]. Unlike the Shapley Additive Explanations, this is a model-specific mea- sure that retrieves the attribution signals spread throughout the input sequence, and therefore is affected by the wider contextual interactions. 3.7 Explainability Evaluation Protocol In line with previous interpretability research [11, 15], a qualitative, case-based evaluation approach is favored over fully automated faithfulness metrics. Repre- sentative cases from the different prediction classes, true positives, false positives and false negatives, are compared to the explanation behavior of the model. This allows the detection of salient explanation patterns and failure modes that can be directly connected to real-life moderation scenarios. 4 Results 4.1 Classification Performance The performance of the RoBERTa-based classifier is first evaluated on the previ- ously mentioned Civil Comments test set. Table 1 summarizes the classification results using accuracy, precision, recall, and F1-score for the toxic class, along with the area under the receiver operating characteristic curve. Table 1. Classification performance on the Civil Comments test set. MetricValue Accuracy0.9405 AUC0.9371 Precision (Toxic)0.6242 Recall (Toxic)0.6203 F1-score (Toxic)0.6222 Although training is limited to a single epoch, the model attains strong overall performance, achieving an accuracy of 0.94 and an AUC of 0.94. These results are consistent with prior work demonstrating that pretrained transformer models provide effective baselines for harmful content detection [1, 9]. Performance on the toxic class is comparatively lower, with an F1-score of 0.62, highlighting the challenges posed by class imbalance and by context-dependent or implicitly expressed toxicity [5, 2]. The objective of this study is not to optimize classification performance, but to establish a reliable baseline for explainability-focused analysis. The observed performance ensures that subsequent interpretation results are grounded in the behavior of a competitive moderation model, rather than artifacts of an under- performing system. Explainability-Driven Analysis of Harmful Content Detection7 4.2 Confusion Matrix Analysis Figure 2 presents the confusion matrix for binary toxicity classification on the test set. The model correctly predicts 3,566 examples out of the 4,000 test sam- ples. The total false positives (118) and false negatives (120) are almost same in number, which shows that the model does not bias toward either over-flagging or under-detecting toxic content. Fig. 2. Confusion matrix for binary harmful content classification on the Civil Com- ments test set. A neutral error pattern forms an optimal rationale for structured and com- parative explanation behavior assessment across prediction groups. The false positive and false negative types, in particular, are vastly different moderation threads – undue moderation versus unfiltered moderation of harmful content. This makes them particularly significant for the human explainability assess- ment in human-in-the-loop [14] scenarios. 4.3 Error Category Breakdown To facilitate a fine-grained analysis of the behavior of the model, its predic- tions on the test set are further classified into four classes: true positives, true negatives, false positives, and false negatives. The distribution is illustrated in Figure 2. False positives are comments with no harassment but politically af- filiated or emotionally expressive, which strongly lexicalize the relevant signals. In turn, false negatives represent comments with indirect, rhetorical, or context- dependent toxicity, which indicate the harmfulness of the content without being 8T. Dhara and S. Sheth explicit about it. Such patterns are in line with the previous research demon- strating that the surface lexicon is not always helpful for the detection of more subtle forms of toxicity [5, 10]. 4.4 Implications for Explainability Analysis The quantitative findings call for a deeper analysis of the model explanations. Although performance scores achieved in the aggregate provide evidence of ef- fective generalization and high predictive power, they do not help to understand why predictions have failed in particular scenarios. To this end, post-hoc explain- ability methods, as described in the analysis pipeline of Figure 1 are resorted to representative examples for each error type to reveal any possible systematic attribution trends and recurring failure scenarios which are missed by standard analysis procedures. 5 Explainability Analysis To characterize model behavior beyond summary metrics, post-hoc explainabil- ity methods are applied at the level of individual predictions. Shapley Additive Explanations and Integrated Gradients are used to generate token-level attribu- tion scores for representative examples drawn from true positive, false positive, and false negative cases, following the pipeline shown in Figure 1. A confidence- aware selection strategy is employed to focus the analysis on systematic model behavior rather than borderline or ambiguous predictions. 5.1 Shapley Additive Explanations Shapley Additive Explanations provide sparse, token-level attributions that high- light the most influential lexical cues contributing to the model’s prediction. For positive, correctly classified toxic examples, Shapley Additive Explanations al- ways provided strong positive attribution scores to explicit insult terms and emotionally-laden vocabulary. An example can be seen in Figure 3 where a mi- nority of tokens drove the model’s prediction, leading to intuitive and inter- pretable explanations. But Shapley Additive Explanations introduce serious weaknesses on misclas- sified samples. For the false positive examples, the method occasionally high- lights the contributions of single lexical triggers (i.e. negative words) explicitly such as swearing or political jargon, even if the triggers contained in the text do not constitute toxic behavior within the context. As presented in image 4, a SHAP approach can misclassify the example as toxic token by token, and the model behaves overconfidently and over-standardized too many examples in less aggressive situations. Explainability-Driven Analysis of Harmful Content Detection9 Fig. 3. Shapley Additive Explanations for a correctly classified toxic comment. Fig. 4. Shapley Additive Explanations for an incorrect prediction (False Positive) 5.2 Integrated Gradients Explanations In contrast to Shapley Additive Explanations, Integrated Gradients produced more distributed attribution profiles which take into account cumulative contri- butions along the input. Rather than focusing many of the tokens of input data that have the most significant impact, it also relatively weights a broader vari- ety of contextual and structural features. Its ability to do so is demonstrated in Table 2, in which there is increased sensitivity to longer-range dependencies for longer comments and in the case of implicitly toxic comments. Table 3 provides 10T. Dhara and S. Sheth a qualitative comparison of the two attribution methods, which illustrates their differing profiles among a representative set of predictions. Table 2. Top Integrated Gradients token attributions for a correctly classified non-toxic comment. The model predicts the non-toxic class with high confidence (P(non-toxic) = 0.997). TokenIG Attribution away0.221 war0.195 to0.165 firms0.105 suspect0.104 position0.100 http0.087 Energy0.066 Table 3. Qualitative comparison of Shapley Additive Explanations and Integrated Gradients attributions on representative prediction outcomes. Example TypeDominant Attribution Pattern Observed Behavior True PositiveDistributed across multiple tokens Contextual cues collectively sup- port correct toxic classification False PositiveConcentrated on isolated lexical cue Over-reliance on single trigger leads to misclassification False NegativeWeak and diffuse attributionSubtle or implicit toxicity not suffi- ciently captured Long CommentsBroad attribution spanCaptures long-range dependencies better than SHAP Short CommentsNarrow attribution spanLimited context reduces explana- tory clarity Additionally, the diffuse quality of the Integrated Gradients explanations may decrease their interpretability for human beings since no one token is obviously the primary driver of the model prediction. This indicates a potential drawback of this approach in terms of the clarity of the explanations provided for the model decisions, in exchange for greater contextual sensitivity. 6 Discussion and Conclusion This work conducted an explainability-driven analysis of a neural harmful con- tent detection model. This analysis is designed to provide insights into model Explainability-Driven Analysis of Harmful Content Detection11 behavior beyond overall performance metrics and highlights how explainability methods can provide insights into important patterns and shortcomings that are not clear from standard evaluation. Despite the impressively high overall accu- racy produced by the RoBERTa-based classifier, an analysis based on Shapley Additive Explanations has revealed that model predictions closely follow some explicit lexical signals. This behavior results in intuitive explanations in the cor- rectly classified instances, but may also be responsible for misclassifications in certain subtle cases. Integrated Gradients allows to place a model attribution in a different light, distributing attributions over a larger number of tokens, while benefiting from contextual information potentially lost in sparser attributions. This is not without its drawbacks, however: increased contextual sensitivity neg- atively impacts interpretability because explanations become less concise and harder for human evaluators to interpret. All these findings suggest a trade-off between context coverage and clarity of explanation through post-hoc explain- ability methods. All in all, the findings suggest that explainable artificial intelli- gence can serve as a valuable mechanism for auditing and diagnosing malicious content moderation systems. By making systematic failure modes, such as lexical over-attribution and challenges in identifying indirect toxicity, visible, explana- tions may aid human-in-the-loop moderation and signal potential avenues for future model enhancement. This study highlights the importance of embedding explainability into the evaluation of intelligent moderation systems in order to further transparency, trustworthiness, and responsible deployment. 6.1 Limitations There are some limitations for this study. Firstly, the proposed analysis is lim- ited to one dataset and a binary classification formulation for harmful content moderation, which may not fully capture the complexity of multi-label harmful content moderation scenarios. Secondly, post-hoc model explanation approaches are sensitive to the specific tokenization, input perturbation and model non- determinism, all of which result in variability for example attribution results. Thirdly, the evaluation emphasizes qualitative interpretability rather than quan- titative faithfulness metrics. While this is justified for the diagnosed objectives taken in this study, it could be a limiting factor nonetheless. For future work, it constitutes an important step to tackle these limitations. Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article. References 1. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019) 2. Mozafari, A., Farahbakhsh, R., Crespi, N.: A BERT-based transfer learning ap- proach for hate speech detection. SN Computer Science 1(6) (2020) 12T. Dhara and S. Sheth 3. Lundberg, S.M., Lee, S.-I.: A unified approach to interpreting model predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS), p. 4765–4774 (2017) 4. Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: Proceedings of the 34th International Conference on Machine Learning (ICML), p. 3319–3328 (2017) 5. Borkan, D., Dixon, L., Sorensen, J., Thain, N., Vasserman, L.: Nuanced metrics for measuring unintended bias with real data for text classification. In: Proceedings of The Web Conference (W), p. 491–500 (2019) 6. Davidson, T., Warmsley, D., Macy, M., Weber, I.: Automated hate speech detection and the problem of offensive language. In: Proceedings of the 11th International AAAI Conference on Web and Social Media (ICWSM), p. 512–515 (2017) 7. Waseem, Z., Hovy, D.: Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter. In: Proceedings of NAACL-HLT, p. 88–93 (2016) 8. Burnap, P., Williams, M.L.: Cyber hate speech on Twitter: An application of ma- chine classification and statistical modeling for policy and decision making. EPJ Data Science 4(1) (2015) 9. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL- HLT, p. 4171–4186 (2019) 10. Dixon, L., Li, J., Sorensen, J., Thain, N., Vasserman, L.: Measuring and miti- gating unintended bias in text classification. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), p. 67–73 (2018) 11. Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Mining (KDD), p. 1135– 1144 (2016) 12. Arras, L., Horn, F., Montavon, G., Müller, K.-R., Samek, W.: Explaining predic- tions of nonlinear classifiers in NLP. In: Proceedings of EMNLP, p. 2791–2797 (2017) 13. Vig, J.: A multiscale visualization of attention in the transformer model. In: Pro- ceedings of the 57th Annual Meeting of the ACL (System Demonstrations), p. 37– 42 (2019) 14. Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P., Inkpen, K., et al.: Guidelines for human-AI interaction. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, p. 1–13 (2019) 15. Doshi-Velez, F., Kim, B.: Towards a rigorous science of interpretable machine learn- ing. arXiv preprint arXiv:1702.08608 (2017)