73 votes
Accepted

Why do we need explainable AI?

As argued by Selvaraju et al., there are three stages of AI evolution, in which interpretability is helpful. In the early stages of AI development, when AI is weaker than human performance, ...
user avatar
  • 3,083
68 votes
Accepted

Do scientists know what is happening inside artificial neural networks?

There are many approaches that aim to make a trained neural network more interpretable and less like a "black box", specifically convolutional neural networks that you've mentioned. Visualizing the ...
user avatar
  • 1,847
32 votes

Do scientists know what is happening inside artificial neural networks?

It depends on what you mean by "know what is happening". Conceptually, yes: ANN perform nonlinear regression. The actual expression represented by the weight matrix/activation function(s) of an ANN ...
user avatar
24 votes
Accepted

Which explainable artificial intelligence techniques are there?

Explainable AI and model interpretability are hyper-active and hyper-hot areas of current research (think of holy grail, or something), which have been brought forward lately not least due to the (...
user avatar
20 votes

Do scientists know what is happening inside artificial neural networks?

Short answer is no. Model interpretability is a hyper-active and hyper-hot area of current research (think of holy grail, or something), which has been brought forward lately not least due to the (...
user avatar
17 votes

Why do we need explainable AI?

Why do we need explainable AI? ... why we need to know "how does its intelligence work?" Because anyone with access to the equipment, enough skill, and enough time, can force the system to make a ...
user avatar
  • 384
12 votes

Do scientists know what is happening inside artificial neural networks?

I'm afraid I don't have the specific citations handy, but I have seen/heard quotes by experts like Andrew Ng and Geoffrey Hinton where they clearly say that we do not really understand neural networks....
user avatar
  • 3,679
9 votes

Why do we need explainable AI?

If you're a bank, hospital or any other entity that uses predictive analytics to make a decision about actions that have huge impact on people's lives, you would not make important decisions just ...
user avatar
9 votes

Do scientists know what is happening inside artificial neural networks?

Not sure if this is what you are searching for, but google extracted images from networks when they were fed with white noise. See Inceptionism: Going Deeper into Neural Networks (Google Research ...
user avatar
  • 199
8 votes
Accepted

How would one debug, understand or fix the outcome of a neural network?

If the observation that the neural network saw was recorded, then yes the prediction can be explained. There was a paper written fairly recently on this topic called "Why Should I Trust You?": ...
user avatar
8 votes
Accepted

Why does nobody use decision trees for visual question answering?

For vision tasks, neural network models almost always include a number of layers that pool and convolute. The convolutions, in particular, are very useful - they can make the model generalize better ...
user avatar
7 votes

Why do we need explainable AI?

Explainable AI is often desirable because AI (in particular, artificial neural networks) can catastrophically fail to do their intended job. More specifically, it can be hacked or attacked with ...
user avatar
  • 32.9k
5 votes

Which explainable artificial intelligence techniques are there?

There are a few XAI techniques that are (partially) agnostic to the model to be interpreted Layer-wise relevance propagation (LRP), introduced in On Pixel-Wise Explanations for Non-Linear Classifier ...
user avatar
  • 32.9k
5 votes

Do scientists know what is happening inside artificial neural networks?

Here is an answer by Carlos E. Perez to the question What is theory behind deep learning? [...] The underlying mathematics of Deep Learning has been in existence for several decades, however ...
user avatar
4 votes

Why do we need explainable AI?

The answer to this is incredibly simple. If you are a bank executive one day you may need to stand up in court and explain why your AI denied mortgages to all these people... who just happen to share ...
user avatar
  • 162
4 votes

Why do we need explainable AI?

Another reason: In the future, AI might be used for tasks that are not possible to be understood by human beings, by understanding how given AI algorithm works on that problem we might understand the ...
user avatar
  • 1,029
4 votes

Do scientists know what is happening inside artificial neural networks?

Do scientists know what is happening inside artificial neural networks? YES Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at ...
user avatar
  • 1,007
3 votes

How is the "right to explanation" reasonable?

How is the right to explanation reasonable, given the current standards at which we hold each other accountable? In short, it is quite reasonable. More specifically, making AI accountable and ...
user avatar
2 votes

Is tabular Q-learning considered interpretable?

There is not a widely accepted definition of explainable AI (XAI). However, as a rule of thumb (my rule of thumb), if you can't explain it easily to a layperson (or even an expert), then the model or ...
user avatar
  • 32.9k
2 votes
Accepted

Is explainable AI more feasible through symbolic AI or soft computing?

XAI is relevant to "black box" AI (machine learning methods where the decision making rationale is not apparent, only the structure of the system that led to that decision.) Symbolic AI, GOFAI, and ...
user avatar
  • 6,077
2 votes

What do the neural network's weights represent conceptually?

I don't know if my intuition is correct but I will give it a try. You could see weights as how much important one thing is, the problem is to understand what that thing represents. When I say thing ...
user avatar
2 votes

How is the "right to explanation" reasonable?

In the paper Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For, the authors claim that the "right to explanation" is unlikely to provide a complete ...
user avatar
  • 32.9k
2 votes

Why are tree-based models more widely used in Medical Diagnosis?

One possible reason may have something to do with the scrutability of models, as described in the first few paragraphs of this article. It presents a case study of a hospital whose policy was to send ...
user avatar
2 votes
Accepted

What exactly is an interpretable machine learning model?

In a simple linear model of the form $y = \beta_0 + \beta_1 x $ we can see that increasing $x$ by a unit will increase the prediction on $y$ by $\beta_1$. Here we can completely determine what the ...
user avatar
2 votes
Accepted

What do the notations $\sim$ and $\Delta (A) $ mean in the paper "Fairness Through Awareness"?

The $\sim$ symbol means that a random variable is drawn from the given distribution, i.e. if I were to say $X$ has a Standard Normal distribution I would write $X \sim \text{Normal}(0,1)$. They write ...
user avatar
2 votes

What needs to be done to make a fair algorithm?

The paper Fair and Unbiased Algorithmic Decision Making: Current State and Future Challenges argues that ensuring fairness is not a trivial task and that the current statistical formalizations of ...
user avatar
  • 32.9k
2 votes

Why do we need explainable AI?

In addition to all these answers mentioning the more practical reasons of why we'd want explainable AIs, I'd like to add a more philosophical one. Understanding how things around us work is one of ...
user avatar
  • 191
2 votes

Are these visualisations the filters of the convolution layer or the convolved images with the filters?

Only the first convolutional layer, with filters that process the input [colour] channels directly, can be rendered directly as image patches in the same domain as the input. The left-most panel in ...
user avatar
  • 23.1k
1 vote

Why are CNN binary classifier output probability distributions often skewed?

Yes, due to this issue, you should use temperature scaling after training your model. It will calibrate your probability and you will start to get the same kind of distributions. Here are a good ...
user avatar
1 vote
Accepted

In GradCAM, why is activation strength considered an indicator of relevant regions?

I think you are misreading the relevant passage here. Since you do not specify exact excerpt(s), I take that by "implicit assumption" you refer to the equation (2) (application of a ReLU) ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible