68

As argued by Selvaraju et al., there are three stages of AI evolution, in all of which interpretability is helpful. In the early stages of AI development, when AI is weaker than human performance, transparency can help us build better models. It can give a better understanding of how a model works and helps us answer several key questions. For example why ...


61

There are many approaches that aim to make a trained neural network more interpretable and less like a "black box", specifically convolutional neural networks that you've mentioned. Visualizing the activations and layer weights Activations visualization is the first obvious and straight-forward one. For ReLU networks, the activations usually start out ...


29

It depends on what you mean by "know what is happening". Conceptually, yes: ANN perform nonlinear regression. The actual expression represented by the weight matrix/activation function(s) of an ANN can be explicitly expanded in symbolic form (e.g. containing sub-expressions such as $1/1+e^{1/1+e^{\dots}}$). However, if by 'know' you mean predicting the ...


15

Why do we need explainable AI? ... why we need to know "how does its intelligence work?" Because anyone with access to the equipment, enough skill, and enough time, can force the system to make a decision that is unexpected. The owner of the equipment, or 3rd parties, relying on the decision without an explanation as to why it is correct would be at a ...


15

Short answer is no. Model interpretability is a hyper-active and hyper-hot area of current research (think of holy grail, or something), which has been brought forward lately not least due to the (often tremendous) success of deep learning models in various tasks; these models are currently only black boxes, and we naturally feel uncomfortable about it... ...


9

I'm afraid I don't have the specific citations handy, but I have seen/heard quotes by experts like Andrew Ng and Geoffrey Hinton where they clearly say that we do not really understand neural networks. That is, we understand something of the how they work (for example, the math behind back propagation) but we don't really understand why they work. It's ...


9

Not sure if this is what you are searching for, but google extracted images from networks when they were fed with white noise. See Inceptionism: Going Deeper into Neural Networks (Google Research Blog). This kind of represents what the network knows.


8

If you're a bank, hospital or any other entity that uses predictive analytics to make a decision about actions that have huge impact on people's lives, you would not make important decisions just because Gradient Boosted trees told you to do so. Firstly, because it's risky and the underlying model might be wrong and, secondly, because in some cases it is ...


7

If the observation that the neural network saw was recorded, then yes the prediction can be explained. There was a paper written fairly recently on this topic called "Why Should I Trust You?": Explaining the Predictions of Any Classifier (2016). In this paper, the author described an algorithm called LIME which is able to explain any machine learning ...


5

Explainable AI is often desirable because AI (in particular, artificial neural networks) can catastrophically fail to do their intended job. More specifically, it can be hacked or attacked with adversarial examples or it can take unexpected wrong decisions whose consequences are catastrophic (for example, it can lead to the death of people). For instance, ...


4

Here is an answer by Carlos E. Perez to the question What is theory behind deep learning? [...] The underlying mathematics of Deep Learning has been in existence for several decades, however the impressive results that we see today are part a consequence of much faster hardware, more data and incremental improvements in methods. Deep Learning ...


4

The answer to this is incredibly simple. If you are a bank executive one day you may need to stand up in court and explain why your AI denied mortgages to all these people... who just happen to share some protected characteristic under anti-discrimination legislation. The judge will not be happy if you handwave the question away mumbling something about ...


4

Another reason: In the future, AI might be used for tasks that are not possible to be understood by human beings, by understanding how given AI algorithm works on that problem we might understand the nature of the given phenomenon.


3

There are a few XAI techniques that are (partially) agnostic to the model to be interpreted Layer-wise relevance propagation (LRP), introduced in On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation (2015) Local Interpretable Model-agnostic Explanations (LIME), introduced in "Why Should I Trust You?" Explaining ...


3

How is the right to explanation reasonable, given the current standards at which we hold each other accountable? In short, it is quite reasonable. More specifically, making AI accountable and responsible for explaining the decision seems reasonable because Humans (DARPA in this case) has chosen to create, raise and evolve AI with the tax-payers money. In ...


2

Do scientists know what is happening inside artificial neural networks? YES Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant? I guess "to know from the kitchen" means "to know in detail"? Let me give you a series of analogies: Does ...


2

There is not a widely accepted definition of explainable AI (XAI). However, as a rule of thumb (my rule of thumb), if you can't explain it easily to a layperson (or even an expert), then the model or algorithm is not (very) interpretable. There are other concepts related to XAI, such as accountability (who is responsible for what?), transparency and fairness....


2

The paper Fair and Unbiased Algorithmic Decision Making: Current State and Future Challenges argues that ensuring fairness is not a trivial task and that the current statistical formalizations of fairness lead to a long list of criteria that are each flawed (or even harmful) in different contexts, that is, there are trade-offs between the proposed ...


2

In addition to all these answers mentioning the more practical reasons of why we'd want explainable AIs, I'd like to add a more philosophical one. Understanding how things around us work is one of the main driving forces of science from antiquity. If you don't have an understanding of how things work, you can't evolve beyond that point. Just because "...


2

The $\sim$ symbol means that a random variable is drawn from the given distribution, i.e. if I were to say $X$ has a Standard Normal distribution I would write $X \sim \text{Normal}(0,1)$. They write two explicit expectations here because $a$ is a random variable with distribution $\mu_x$ but $X$ is also a random variable with distribution $V$. I believe you ...


1

There are many frameworks which allow you to do that. One of them, which supports many different techniques for visualization, can be found here: https://github.com/marcoancona/DeepExplain


1

Yes there definitely is, and research into this has actually resulted in some really cool behaviour. One of the simplest ways is to simply back propagate the gradient all the way back to the input. Areas of the input that affected the final decision will receive larger gradients. Interestingly, this also sort of works as a rudimentary form of semantic ...


1

XAI is relevant to "black box" AI (machine learning methods where the decision making rationale is not apparent, only the structure of the system that led to that decision.) Symbolic AI, GOFAI, and Expert Systems are both explainable and understood, in that the the decision-making process is designed by humans. (Symbolic AI involves human-readable ...


Only top voted, non community-wiki answers of a minimum length are eligible