Which explainable artificial intelligence techniques are there?

Question

Explainable artificial intelligence (XAI) is concerned with the development of techniques that can enhance the interpretability, accountability, and transparency of artificial intelligence and, in particular, machine learning algorithms and models, especially black-box ones, such as artificial neural networks, so that these can also be adopted in areas, like healthcare, where the interpretability and understanding of the results (e.g. classifications) are required.

Which XAI techniques are there?

If there are many, to avoid making this question too broad, you can just provide a few examples (the most famous or effective ones), and, for people interested in more techniques and details, you can also provide one or more references/surveys/books that go into the details of XAI. The idea of this question is that people could easily find one technique that they could study to understand what XAI really is or how it can be approached.

Answer 1

Explainable AI and model interpretability are hyper-active and hyper-hot areas of current research (think of holy grail, or something), which have been brought forward lately not least due to the (often tremendous) success of deep learning models in various tasks, plus the necessity of algorithmic fairness & accountability.

Here are some state of the art algorithms and approaches, together with implementations and frameworks.

---

Model-agnostic approaches

• LIME: Local Interpretable Model-agnostic Explanations (paper, code, blog post, R port)

• SHAP: A Unified Approach to Interpreting Model Predictions (paper, Python package, R package). GPU implementation for tree models by NVIDIA using RAPIDS - GPUTreeShap (paper, code, blog post)

• Anchors: High-Precision Model-Agnostic Explanations (paper, authors' Python code, Java implementation)

• Diverse Counterfactual Explanations (DiCE) by Microsoft (paper, code, blog post)

• Black Box Auditing and Certifying and Removing Disparate Impact (authors' Python code)

• FairML: Auditing Black-Box Predictive Models, by Cloudera Fast Forward Labs (blog post, paper, code)

SHAP seems to enjoy high popularity among practitioners; the method has firm theoretical foundations on co-operational game theory (Shapley values), and it has in a great degree integrated the LIME approach under a common framework. Although model-agnostic, specific & efficient implementations are available for neural networks (DeepExplainer) and tree ensembles (TreeExplainer, paper).

---

Neural network approaches (mostly, but not exclusively, for computer vision models)

• The Layer-wise Relevance Propagation (LRP) toolbox for neural networks (2015 paper @ PLoS ONE, 2016 paper @ JMLR, project page, code, TF Slim wrapper)

• Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (paper, authors' Torch code, Tensorflow code, PyTorch code, yet another Pytorch implementation, Keras example notebook, Coursera Guided Project)

• Axiom-based Grad-CAM (XGrad-CAM): Towards Accurate Visualization and Explanation of CNNs, a refinement of the existing Grad-CAM method (paper, code)

• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability (paper, code, Google blog post)

• TCAV: Testing with Concept Activation Vectors (ICML 2018 paper, Tensorflow code)

• Integrated Gradients (paper, code, Tensorflow tutorial, independent implementations)

• Network Dissection: Quantifying Interpretability of Deep Visual Representations, by MIT CSAIL (project page, Caffe code, PyTorch port)

• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, by MIT CSAIL (project page, with links to paper & code)

• Explain to Fix: A Framework to Interpret and Correct DNN Object Detector Predictions (paper, code)

• Transparecy-by-Design (TbD) networks (paper, code, demo)

• Distilling a Neural Network Into a Soft Decision Tree, a 2017 paper by Geoff Hinton, with various independent PyTorch implementations

• Understanding Deep Networks via Extremal Perturbations and Smooth Masks (paper), implemented in TorchRay (see below)

• Understanding the Role of Individual Units in a Deep Neural Network (preprint, 2020 paper @ PNAS, code, project page)

• GNNExplainer: Generating Explanations for Graph Neural Networks (paper, code)

• Benchmarking Deep Learning Interpretability in Time Series Predictions (paper @ NeurIPS 2020, code utilizing Captum)

• Concept Whitening for Interpretable Image Recognition (paper, preprint, code)

---

Libraries & frameworks

As interpretability moves toward the mainstream, there are already frameworks and toolboxes that incorporate more than one of the algorithms and techniques mentioned and linked above; here is a partial list:

• The ELI5 Python library (code, documentation)

• DALEX - moDel Agnostic Language for Exploration and eXplanation (homepage, code, JMLR paper), part of the DrWhy.AI project

• The What-If tool by Google, a feature of the open-source TensorBoard web application, which let users analyze an ML model without writing code (project page, code, blog post)

• The Language Interpretability Tool (LIT) by Google, a visual, interactive model-understanding tool for NLP models (project page, code, blog post)

• Lucid, a collection of infrastructure and tools for research in neural network interpretability by Google (code; papers: Feature Visualization, The Building Blocks of Interpretability)

• TorchRay by Facebook, a PyTorch package implementing several visualization methods for deep CNNs

• iNNvestigate Neural Networks (code, JMLR paper)

• tf-explain - interpretability methods as Tensorflow 2.0 callbacks (code, docs, blog post)

• InterpretML by Microsoft (homepage, code still in alpha, paper)

• Captum by Facebook AI - model interpetability for Pytorch (homepage, code, intro blog post)

• Skater, by Oracle (code, docs)

• Alibi, by SeldonIO (code, docs)

• AI Explainability 360, commenced by IBM and moved to the Linux Foundation (homepage, code, docs, IBM Bluemix, blog post)

• Ecco: explaining transformer-based NLP models using interactive visualizations (homepage, code, article).

• Recipes for Machine Learning Interpretability in H2O Driverless AI (repo)

---

Reviews & general papers

• A Survey of Methods for Explaining Black Box Models (2018, ACM Computing Surveys)

• Definitions, methods, and applications in interpretable machine learning (2019, PNAS)

• Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead (2019, Nature Machine Intelligence, preprint)

• Machine Learning Interpretability: A Survey on Methods and Metrics (2019, Electronics)

• Principles and Practice of Explainable Machine Learning (2020, preprint)

• Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges (keynote at 2020 ECML XKDD workshop by Christoph Molnar, video & slides)

• Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020, Information Fusion)

• Counterfactual Explanations for Machine Learning: A Review (2020, preprint, critique by Judea Pearl)

• Interpretability 2020, an applied research report by Cloudera Fast Forward, updated regularly

• Interpreting Predictions of NLP Models (EMNLP 2020 tutorial)

• Explainable NLP Datasets (site, preprint, highlights)

• Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges

---

eBooks (available online)

• Interpretable Machine Learning, by Christoph Molnar, with R code available

• Explanatory Model Analysis, by DALEX creators Przemyslaw Biecek and Tomasz Burzykowski, with both R & Python code snippets

• An Introduction to Machine Learning Interpretability (2nd ed. 2019), by H2O

---

Online courses & tutorials

• Machine Learning Explainability, Kaggle tutorial

• Explainable AI: Scene Classification and GradCam Visualization, Coursera guided project

• Explainable Machine Learning with LIME and H2O in R, Coursera guided project

• Interpretability and Explainability in Machine Learning, Harvard COMPSCI 282BR

---

Other resources

• explained.ai blog

• A Twitter thread, linking to several interpretation tools available for R

• A whole bunch of resources in the Awesome Machine Learning Interpetability repo

• The online comic book (!) The Hitchhiker's Guide to Responsible Machine Learning, by the team behind the textbook Explanatory Model Analysis and the DALEX package mentioned above (blog post and backstage)

Answer 2

There are a few XAI techniques that are (partially) agnostic to the model to be interpreted

• Layer-wise relevance propagation (LRP), introduced in On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation (2015)

• Local Interpretable Model-agnostic Explanations (LIME), introduced in "Why Should I Trust You?" Explaining the Predictions of Any Classifier (2016)

• Model Agnostic Contrastive Explanations Method (MACEM), introduced in Model Agnostic Contrastive Explanations for Structured Data (2019)

There are also ML models that are not considered black boxes and that are thus more interpretable than black boxes, such as

• linear models (e.g. linear regression)
• decision trees
• naive Bayes (and, in general, Bayesian networks)

For a more complete list of such techniques and models, have a look at the online book Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, by Christoph Molnar, which attempts to categorise and present the main XAI techniques.

Answer 3

Jacques Pitrat's last book Artificial Beings The Conscience of a Conscious Machine has some interesting chapters (§2, §3, §5, §6, §7) related to the questions of explanations and meta-explanations in AI systems.

It describes a reflexive, "expert system" like, meta-knowledge based approach to explainable AI

You could also read his papers Implementation of a reflective system (1996) and A Step toward an Artificial AI Scientist online (there could be a typo in it: "pile" is the French word for stack, including the call stack).

You might also look into J.Pitrat's blog and into the ongoing RefPerSys project (it is in November 2020).

PS. J.Pitrat (born in 1934) passed away in October 2019. French readers could see this. His blog might disappear in a few months.