If the original purpose for developing AI was to help humans in some tasks and that purpose still holds, why should we care about its explainability? For example, in deep learning, as long as the intelligence helps us to the best of their abilities and carefully arrives at its decisions, why would we need to know how its intelligence works?

  • 15
    $\begingroup$ What happens when the AI doesn't give you a good answer? How do you find the problem? How do you verify the solution, if you don't understand the reasoning? How do you make sure we don't get all turned into paperclips? :) $\endgroup$ – Luaan Sep 2 '19 at 8:57
  • 8
    $\begingroup$ Not posting it as an answer since I suspect it doesn't meet the standards, but suppose we ask an AI how to fix climate issues and it says "pollute the oceans". Maybe it's right and there's some contrived way in which that fixes things. Maybe a developer made an error and the actual output should have been "unpollute the oceans". How do you intend to distinguish between the two, if not by veryfing the AI's reasoning? $\endgroup$ – Flater Sep 2 '19 at 10:34
  • 5
    $\begingroup$ Is any answer to this question really needed, beyond the common knowledge of the existence of adversarial techniques? Right now it's possible to make tiny tweaks to input data that create massively disproportionate changes in the output. In tasks such as visual recognition of objects, in which the results can be easily checked against a real intelligence, the results appear to be nonsensical and insane, and indicate that what the AI is "seeing" is something very different from what we are seeing. If the AI is unable to explain itself in such cases, its usefulness drops off sharply. $\endgroup$ – Mason Wheeler Sep 2 '19 at 19:41
  • 1
    $\begingroup$ While your question is based on the need for explainable AI, I would also like to propose a reading which challenges this view - I think there are very good points there. Among other things, one reason for the need of explainability might be linked to our human need for explaining the systems around us, and then our higher trust in systems that we understand, even if these systems underperform others that we do NOT understand. Hope it adds some material for reflection - hackernoon.com/… $\endgroup$ – Elisio Quintino Sep 3 '19 at 8:30
  • 2
    $\begingroup$ This might be an interesting additional resoruce. People tried to build a Husky vs Wolf classifier, then realized that the network didn't focus on the animal, but on the background because all images with snow in the background were wolves. In other words, if you detect a car, you need to be sure that that's because of a car in the image and not because of a stop sign on the side of the image. $\endgroup$ – jaaq Sep 4 '19 at 10:53

As argued by Selvaraju et al., there are three stages of AI evolution, in all of which interpretability is helpful.

  1. In the early stages of AI development, when AI is weaker than human performance, transparency can help us build better models. It can give a better understanding of how a model works and helps us answer several key questions. For example why a model works in some cases and doesn't in others, why some examples confuse the model more than others, why these types of models work and the others don't, etc.

  2. When AI is on par with human performance and ML models are starting to be deployed in several industries, it can help build trust for these models. I'll elaborate a bit on this later, because I think that it is the most important reason.

  3. When AI significantly outperforms humans (e.g. AI playing chess or Go), it can help with machine teaching (i.e. learning from the machine on how to improve human performance on that specific task).

Why is trust so important?

First, let me give you a couple of examples of industries where trust is paramount:

  • In healthcare, imagine a Deep Neural Net performing diagnosis for a specific disease. A classic black box NN would just output a binary "yes" or "no". Even if it could outperform humans in sheer predictability, it would be utterly useless in practice. What if the doctor disagreed with the model's assessment, shouldn't he know why the model made that prediction; maybe it saw something the doctor missed. Furthermore, if it made a misdiagnosis (e.g. a sick person was classified as healthy and didn't get the proper treatment), who would take responsibility: the model's user? the hospital? the company that designed the model? The legal framework surrounding this is a bit blurry.

  • Another example are self-driving cars. The same questions arise: if a car crashes whose fault is it: the driver's? the car manufacturer's? the company that designed the AI? Legal accountability, is key for the development of this industry.

In fact, this lack of trust, has according to many hindered the adoption of AI in many fields (sources: 1, 2, 3). While there is a running hypothesis that with more transparent, interpretable or explainable systems users will be better equipped to understand and therefore trust the intelligent agents (sources: 1, 2, 3).

In several real world applications you can't just say "it works 94% of the time". You might also need to provide a justification...

Government regulations

Several governments are slowly proceeding to regulate AI and transparency seems to be at the center of all of this.

The first to move in this direction is the EU, which has set several guidelines where they state that AI should be transparent (sources: 1, 2, 3). For instance the GDPR states that if a person's data has been subject to "automated decision-making" or "profiling" systems, then he has a right to access

"meaningful information about the logic involved"

(Article 15, EU GDPR)

Now this is a bit blurry, but there is clearly the intent of requiring some form of explainability from these systems. The general idea the EU is trying to pass is that "if you have an automated decision-making system affecting people's lives then they have a right to know why a certain decision has been made." For example a bank has an AI accepting and declining loan applications, then the applicants have a right to know why their application was rejected.

To sum up...

Explainable AIs are necessary because:

  • It gives us a better understanding, which helps us improve them.
  • In some cases we can learn from AI how to make better decisions in some tasks.
  • It helps users trust AI, which which leads to a wider adoption of AI.
  • Deployed AIs in the (not to distant) future might be required to be more "transparent".
  • 4
    $\begingroup$ That's exactly the problem (IMHO). Often you can't test exhaustively a black-box and then you accept (even in life critical applications) something that "seems good enough", in this case the human brain, and we're all good with that. If we can't do it for AIs then it's because of (limitations?) in our regulation, not because "to fly an airplane everything has to be mathematically proven" (it's not, even nowadays without any AI). Of course the way you test an AI cannot be the same way you test a human pilot (even if they can partially coincide). $\endgroup$ – Adriano Repetti Sep 2 '19 at 11:50
  • 11
    $\begingroup$ @ZsoltSzilagy I'm not saying that government regulations are a bad thing. I'm just saying that they are a serious reason to develop explainable AI, if you're working in some industries. $\endgroup$ – Djib2011 Sep 2 '19 at 13:44
  • 8
    $\begingroup$ One other point to add is concentration of risk. A drug-sniffing dog might be biased, and you can't ask it why it made certain decisions. But it's just one dog out of thousands. A single AI model is going to be deployed globally, so if it's wrong it has a much larger impact. $\endgroup$ – Brendan Whiting Sep 2 '19 at 19:14
  • 19
    $\begingroup$ @BrendanWhiting actually that's an interesting case. What's the difference between a drug-sniffing dog and a drug-sniffing AI. Why require the AI to give an explanation while the dog doesn't have to.... I think the difference that in the first case, the dog is just a tool that helps the human (in this case the DEA cop) to make his decision. He is ultimately the one responsible for it, not the dog. Similarly, there is no problem with decision-support AIs, only with decision-making ones. That's how I think this whole accountability thing in many domains will ultimately be bypassed. $\endgroup$ – Djib2011 Sep 2 '19 at 21:14
  • 6
    $\begingroup$ Maybe it's also a problem of unknown unknowns. We've trained animals for thousands of years and we're pretty confident that this is stable. If there were going to be packs of marauding trained animals that took over the world, it probably would have happened by now. (Although I kind of like the theory that most dogs are kind of parasites, they have evolved to be so cute that we take care of them instead of our own young). $\endgroup$ – Brendan Whiting Sep 2 '19 at 21:43

Why do we need explainable AI? ... why we need to know "how does its intelligence work?"

Because anyone with access to the equipment, enough skill, and enough time, can force the system to make a decision that is unexpected. The owner of the equipment, or 3rd parties, relying on the decision without an explanation as to why it is correct would be at a disadvantage.

Examples - Someone might discover:

  • People whom are named John Smith and request heart surgery on: Tuesday mornings, Wednesday afternoons, or Fridays on odd days and months have a 90% chance of moving to the front of the line.

  • Couples whom have the male's last name an odd letter in the first half of the alphabet and apply for a loan with a spouse whose first name begins with a letter from the beginning of the alphabet are 40% more likely to receive the loan if they have fewer than 5 bad entries in their credit history.

  • etc.

Notice that the above examples ought not to be determining factors in regards to the question being asked, yet it's possible for an adversary (with their own equipment, or knowledge of the algorithm) to exploit it.

Source papers:

  • "AdvHat: Real-world adversarial attack on ArcFace Face ID system" (Aug 23 2019) by Stepan Komkov and Aleksandr Petiushko

    • Creating a sticker and placing it on your hat fools facial recognition system.
  • "Defending against Adversarial Attacks through Resilient Feature Regeneration" (Jun 8 2019), by Tejas Borkar, Felix Heide, and Lina Karam

    • "Deep neural network (DNN) predictions have been shown to be vulnerable to carefully crafted adversarial perturbations. Specifically, so-called universal adversarial perturbations are image-agnostic perturbations that can be added to any image and can fool a target network into making erroneous predictions. Departing from existing adversarial defense strategies, which work in the image domain, we present a novel defense which operates in the DNN feature domain and effectively defends against such universal adversarial attacks. Our approach identifies pre-trained convolutional features that are most vulnerable to adversarial noise and deploys defender units which transform (regenerate) these DNN filter activations into noise-resilient features, guarding against unseen adversarial perturbations.".

  • "One pixel attack for fooling deep neural networks" (May 3 2019), by Jiawei Su, Danilo Vasconcellos Vargas, and Sakurai Kouichi

    • Altering one pixel can cause these errors:

    Figure 1
    Fig. 1. One-pixel attacks created with the proposed algorithm that successfully fooled three types of DNNs trained on CIFAR-10 dataset: The All convolutional network (AllConv), Network in network (NiN) and VGG. The original class labels are in black color while the target class labels and the corresponding confidence are given below.


    Figure 2
    Fig. 2. One-pixel attacks on ImageNet dataset where the modified pixels are highlighted with red circles. The original class labels are in black color while the target class labels and their corresponding confidence are given below.

Without an explanation as to how and why a decision is arrived at the decision can't be absolutely relied upon.

  • 1
    $\begingroup$ The findings of one pixel attacks and other similar things is why I claim that deep learning isn't learning anything. $\endgroup$ – Joshua Sep 3 '19 at 15:55
  • 1
    $\begingroup$ @Joshua Well it's not learning as in human learning (understanding the reasoning), but it does learn the multi-dimensional function that can classify these image in categories. It did do that on it's own, even though it did solve for the best solution using brute force.That is a lot to expect from a bundle of metal, and sand arranged neatly to allow electricity to flow in systematic manner. $\endgroup$ – user14492 Sep 4 '19 at 11:50
  • 1
    $\begingroup$ WRT to your examples, say the loan on couples with a funky condition, isn't this kind of thing mitigated by only feeding the model data that it should care about? Like the name should never be fed to the AI, because the AI should never make determinations based on the requester's name. I understand your point though, there could be less contrived examples against data points that the AI does actually need. The one pixel attacks are super interesting $\endgroup$ – Cruncher Sep 4 '19 at 17:34
  • $\begingroup$ @Cruncher Less contrived examples are always better when one is making a point, much as hindsight is 20/20. How about if the previous refused loan had both a prime number within it (larger than 7) and the previous one had two numbers that were prime and between 17 and 43, etc. Then you find someone whom says that they will sell something for X dollars, take out a loan and try to get disqualified, do that a few times, now take out a loan for a house or ship - 30% better chance and prior refusals discounted. Without an explanation behind the reason you would always want to make your own choice. $\endgroup$ – Rob Sep 4 '19 at 18:16
  • $\begingroup$ @Joshua Well, they can certainly learn how to find one pixel attacks and similar things. (The paper linked here used an evolutionary approach, but I've seen similar results using GANs.) No method of classification is going to be perfect. Human vision is vulnerable to optical illusions. This is the machine equivalent. We can just find more extreme examples for machine vision systems because they can make and record predictions more quickly than humans do, so we can effectively train another system to find failure states matching certain criteria. $\endgroup$ – Ray Sep 4 '19 at 21:48

If you're a bank, hospital or any other entity that uses predictive analytics to make a decision about actions that have huge impact on people's lives, you would not make important decisions just because Gradient Boosted trees told you to do so. Firstly, because it's risky and the underlying model might be wrong and, secondly, because in some cases it is illegal - see Right to explanation.


Explainable AI is often desirable because

  1. AI (in particular, artificial neural networks) can catastrophically fail to do their intended job. More specifically, it can be hacked or attacked with adversarial examples or it can take unexpected wrong decisions whose consequences are catastrophic (for example, it can lead to the death of people). For instance, imagine that an AI is responsible for determining the dosage of a drug that needs to be given to a patient, based on the conditions of the patient. What if the AI makes a wrong prediction and this leads to the death of the patient? Who will be responsible for such an action? In order to accept the dosage prediction of the AI, the doctors need to trust the AI, but trust only comes with understanding, which requires an explanation. So, to avoid such possible failures, it is fundamental to understand the inner workings of the AI, so that it does not make those wrong decisions again.

  2. AI often needs to interact with humans, which are sentient beings (we have feelings) and that often need an explanation or reassurance (regarding some topic or event).

  3. In general, humans are often looking for an explanation and understanding of their surroundings and the world. By nature, we are curious and exploratory beings. Why does an apple fall?


The answer to this is incredibly simple. If you are a bank executive one day you may need to stand up in court and explain why your AI denied mortgages to all these people... who just happen to share some protected characteristic under anti-discrimination legislation. The judge will not be happy if you handwave the question away mumbling something about algorithms. Or worse, why did this car/plane crash and how will you prevent it next time.

This is the major blocker to more widespread adoption of AI in many industries.

  • 1
    $\begingroup$ "The answer" is overstating your case. I would upvote if you said "One answer ..." $\endgroup$ – John Coleman Sep 4 '19 at 16:19

Another reason: In the future, AI might be used for tasks that are not possible to be understood by human beings, by understanding how given AI algorithm works on that problem we might understand the nature of the given phenomenon.


In addition to all these answers mentioning the more practical reasons of why we'd want explainable AIs, I'd like to add a more philosophical one.

Understanding how things around us work is one of the main driving forces of science from antiquity. If you don't have an understanding of how things work, you can't evolve beyond that point. Just because "gravity works" hasn't stopped us trying to understand how it works. In turn a better understanding of it led to several key discoveries, which have helped us advance our technology.

Likewise, if we stop at "it works" we will stop improving it.


AI hasn't been just about making "machines think", but also through them to understand how the human brain works. AI and neuroscience go hand-by-hand.

This all wouldn't be possible without being able to explain AI.


It should not be assumed that the development of AI was originally motivated by the desire to help humans. There are many plausible explanations equally difficult to either prove or disprove.

  • Be known to dream up some futuristic idea before anyone else does
  • Acquire power in advance of some imagined enemy or some future potential one
  • Because it might be possible
  • For fun
  • Because the U.S. Department of Defense would probably fund it indefinitely
  • It's a good career move
  • To prove that there is nothing particularly miraculous about human brains
  • We were hired and given some money, and it seemed like a good way to spend it
  • It was decided to pursue it but none of us really remember why

There are some poorly defined descriptive words in this question too, although it may be difficult to find any better words to replace them. How would we formalize these?

  • To their best abilities (the intelligent systems) --- In what way would we gauge abilities and compare results with them? We say to a student, "You're not applying yourself," but that is hardly a scientific observation. It is a somewhat arbitrary judgement based on a projection of achievement that was not met according to the grading system of a third party and its application by other fallible parties.
  • Carefully arriving at decisions --- Care implies objectives that are themselves objective. We do not yet have documented an advance computing platform that encodes a system of ethics applied to an abstracted awareness of situations, such as in the case of an ethical human being, whereby care obtains any realistic meaning. That a nand gate performs a nand function reliably or some algorithm is proven to converge with a given data set size under specific conditions is hardly a fully extended meaning of what we are when we are careful.
  • Explainable --- This is ambiguous too. On one extreme, convergence on a set of parameter values during an artificial network's convergence is an explanation, but the individual values are not explained. On the opposite extreme, a full report of a hypothesis, experimental design, choice of a set of conditions, analysis, results, and conclusion is still not an exhaustive explanation. Such a report may only include below 1% of the information describing the application of human intelligence to the research outlined by the report.

The early work on artificial networks was criticized in AI journals of the early 1990s for not being explainable on the basis of tracability. Production (rule-based) systems left audit trails of rules that were applied and to the results of what previous rules so that someone could assemble a written proof of the result. This was of limited usefulness.

When steering wheels are removed from vehicles and some jurisdictions begin to legislate against human driving in some regions, it will not be because the proofs of safety in a million scenarios were written out. It will be because the distributions of recorded accidental deaths, dismemberments, and destructions of property resulting from an AI driver installed in a particular vehicle type, over a sufficiently convincing period of time, indicates its safety over those distributions for human drivers. Eventually in some court room or legislative caucus someone will say this or its equivalent.

If we don't outlaw human driving under these specified conditions for the region under discussion, we are sentencing X number of men, women, children, and elderly pedestrians and passengers per year to a premature death.

Understanding mechanism of action and the decisions made for specific cases is useful, but why such is useful is as indeterminate as why AI became a viable field of work and study.

  • It would be interesting to compare competitive AI systems in some quantifiable manner.
  • It would be of great academic value to understand more about intelligence.
  • A better explanation makes for a good paper.
  • I was doodling one day and arrived at a way to explain a particular class of systems that seemed to be poorly explained.

Although the non-auditability of AI systems may come up on the floor of legislative and judicial events, much of the decisions made will be on the basis of the way statistical evaluations are published and perceived. Those that insist the systems operate in a way that can be explained will probably, consciously or subconsciously, be motivated by an interest in a perception that the dominance of human beings is manifest destiny. It is more than just ironic that many of those that helped U.S. and U.S.S.R. interests during the Cold War are now considered terrorist leaders by the successors of both Cold War factions.

The related and more clearly answerable question is whether an intelligent helper can be expected to remain a helper indefinitely. The investigation of forever-intelligent-forever-helpers is ongoing and of remarkable interest to everyone from sci fi authors and screenwriters to military affiliate think tankers.


IMHO, the most important need for explainable AI is to prevent us from becoming intellectually lazy. If we stop trying to understand how answers are found, we have conceded the game to our machines.

  • $\begingroup$ Well, we understand how they find answers on a high level: They guess around millions of times until they find a promising pattern. The intellectual task is to understand WHAT CAUSES those patters - a question an AI coulnd't care less about. $\endgroup$ – Zsolt Szilagy Sep 2 '19 at 11:01
  • $\begingroup$ Sometimes that's true, but some types of AI (e.g., genetic algorithms) often do better than that and can be structured to provide good clues re what causes patterns. Sometimes simply pointing out that "the pattern found in this data set is very closely analogous to the pattern found in this other data set" can be very informative and lead to an explanation of cause. $\endgroup$ – S. McGrew Sep 2 '19 at 13:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.