75
votes
Accepted
Why do we need explainable AI?
As argued by Selvaraju et al., there are three stages of AI evolution, in which interpretability is helpful.
In the early stages of AI development, when AI is weaker than human performance, ...
- 3,143
69
votes
Accepted
Do scientists know what is happening inside artificial neural networks?
There are many approaches that aim to make a trained neural network more interpretable and less like a "black box", specifically convolutional neural networks that you've mentioned.
Visualizing the ...
- 1,917
32
votes
Do scientists know what is happening inside artificial neural networks?
It depends on what you mean by "know what is happening".
Conceptually, yes: ANN perform nonlinear regression. The actual expression represented by the weight matrix/activation function(s) of an ANN ...
- 7,176
28
votes
Accepted
Which explainable artificial intelligence techniques are there?
Explainable AI and model interpretability are hyper-active and hyper-hot areas of current research (think of holy grail, or something), which have been brought forward lately not least due to the (...
- 979
20
votes
Do scientists know what is happening inside artificial neural networks?
Short answer is no.
Model interpretability is a hyper-active and hyper-hot area of current research (think of holy grail, or something), which has been brought forward lately not least due to the (...
- 979
17
votes
Why do we need explainable AI?
Why do we need explainable AI?
... why we need to know "how does its intelligence work?"
Because anyone with access to the equipment, enough skill, and enough time, can force the system to make a ...
- 608
12
votes
Do scientists know what is happening inside artificial neural networks?
I'm afraid I don't have the specific citations handy, but I have seen/heard quotes by experts like Andrew Ng and Geoffrey Hinton where they clearly say that we do not really understand neural networks....
- 3,717
9
votes
Why do we need explainable AI?
If you're a bank, hospital or any other entity that uses predictive analytics to make a decision about actions that have huge impact on people's lives, you would not make important decisions just ...
9
votes
Do scientists know what is happening inside artificial neural networks?
Not sure if this is what you are searching for, but google extracted images from networks when they were fed with white noise.
See Inceptionism: Going Deeper into Neural Networks (Google Research ...
- 199
8
votes
Accepted
How would one debug, understand or fix the outcome of a neural network?
If the observation that the neural network saw was recorded, then yes the prediction can be explained.
There was a paper written fairly recently on this topic called "Why Should I Trust You?": ...
- 853
8
votes
Accepted
Why does nobody use decision trees for visual question answering?
For vision tasks, neural network models almost always include a number of layers that pool and convolute. The convolutions, in particular, are very useful - they can make the model generalize better ...
- 416
7
votes
Why do we need explainable AI?
Explainable AI is often desirable because
AI (in particular, artificial neural networks) can catastrophically fail to do their intended job. More specifically, it can be hacked or attacked with ...
- 37k
5
votes
Which explainable artificial intelligence techniques are there?
There are a few XAI techniques that are (partially) agnostic to the model to be interpreted
Layer-wise relevance propagation (LRP), introduced in On Pixel-Wise Explanations for Non-Linear Classifier ...
- 37k
5
votes
Do scientists know what is happening inside artificial neural networks?
Here is an answer by Carlos E. Perez to the question What is theory behind deep learning?
[...]
The underlying mathematics of Deep Learning has been in existence for several decades, however ...
4
votes
Why do we need explainable AI?
The answer to this is incredibly simple. If you are a bank executive one day you may need to stand up in court and explain why your AI denied mortgages to all these people... who just happen to share ...
- 162
4
votes
Why do we need explainable AI?
Another reason: In the future, AI might be used for tasks that are not possible to be understood by human beings, by understanding how given AI algorithm works on that problem we might understand the ...
- 1,240
3
votes
Do scientists know what is happening inside artificial neural networks?
Do scientists know what is happening inside artificial neural networks?
YES
Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at ...
- 1,025
3
votes
How is the "right to explanation" reasonable?
How is the right to explanation reasonable, given the current standards at which we hold each other accountable?
In short, it is quite reasonable.
More specifically, making AI accountable and ...
- 179
3
votes
Are these visualisations the filters of the convolution layer or the convolved images with the filters?
Only the first convolutional layer, with filters that process the input [colour] channels directly, can be rendered directly as image patches in the same domain as the input. The left-most panel in ...
- 26.5k
2
votes
Is tabular Q-learning considered interpretable?
There is not a widely accepted definition of explainable AI (XAI). However, as a rule of thumb (my rule of thumb), if you can't explain it easily to a layperson (or even an expert), then the model or ...
- 37k
2
votes
Accepted
Is explainable AI more feasible through symbolic AI or soft computing?
XAI is relevant to "black box" AI (machine learning methods where the decision making rationale is not apparent, only the structure of the system that led to that decision.)
Symbolic AI, GOFAI, and ...
- 6,177
2
votes
What do the neural network's weights represent conceptually?
I don't know if my intuition is correct but I will give it a try.
You could see weights as how much important one thing is, the problem is to understand what that thing represents. When I say thing ...
- 666
2
votes
How is the "right to explanation" reasonable?
In the paper Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For, the authors claim that the "right to explanation" is unlikely to provide a complete ...
- 37k
2
votes
Why are tree-based models more widely used in Medical Diagnosis?
One possible reason may have something to do with the scrutability of models, as described in the first few paragraphs of this article. It presents a case study of a hospital whose policy was to send ...
- 53
2
votes
What needs to be done to make a fair algorithm?
The paper Fair and Unbiased Algorithmic Decision Making: Current State and Future Challenges argues that ensuring fairness is not a trivial task and that the current statistical formalizations of ...
- 37k
2
votes
Why do we need explainable AI?
In addition to all these answers mentioning the more practical reasons of why we'd want explainable AIs, I'd like to add a more philosophical one.
Understanding how things around us work is one of ...
- 191
2
votes
Accepted
What do the notations $\sim$ and $\Delta (A) $ mean in the paper "Fairness Through Awareness"?
The $\sim$ symbol means that a random variable is drawn from the given distribution, i.e. if I were to say $X$ has a Standard Normal distribution I would write $X \sim \text{Normal}(0,1)$. They write ...
- 4,400
2
votes
Accepted
What exactly is an interpretable machine learning model?
In a simple linear model of the form $y = \beta_0 + \beta_1 x $ we can see that increasing $x$ by a unit will increase the prediction on $y$ by $\beta_1$. Here we can completely determine what the ...
- 4,400
2
votes
How can I interpret the way the neural network is producing an output for a given input?
"Ideally, I would end up with an equation that would allow me to perform the classification without the network".
If you could find such analytic equation without machine learning then why ...
- 4,753
1
vote
Accepted
In GradCAM, why is activation strength considered an indicator of relevant regions?
I think you are misreading the relevant passage here.
Since you do not specify exact excerpt(s), I take that by "implicit assumption" you refer to the equation (2) (application of a ReLU) ...
- 979
Only top scored, non community-wiki answers of a minimum length are eligible