Questions tagged [explainable-ai]
For questions related to explainable artificial intelligence (XAI), also known as interpretable AI, which refers to AI techniques that can be trusted and easily understood by humans, which are particularly relevant in areas like healthcare or self-driving cars. There are several concepts related to XAI, such as accountability, fairness, and transparency.
48
questions
0
votes
0
answers
11
views
LRP (Layer wise relevance propagation ) backward pass for two layer LSTM Networks
I am trying to calculate relevance scores for each time step of the input of my network which is a time series of shape (batch_size = 7000,sequence_length = 20,number of features = 1).
my network has ...
- 1
0
votes
0
answers
16
views
Is there a way an AI could observe patterns in another AI?
I am really interested in “explainable AI” in which you can try to see the factors influencing a model’s decisions and ideally reduce it to a human-understandable explanation of why it does what, when....
- 119
2
votes
1
answer
40
views
Is there any interpretation method suitable for CNNs which do regression tasks?
I mainly tackle regression problems by CNNs, and want to find a reliable method to calculate the heatmaps for NN's results. However, I find almost all interpretation methods including CAM is used for ...
- 21
0
votes
1
answer
194
views
Are there Explainable GNN methods for node regression tasks?
I am wondering if there are gnn explainable methods for a regression task (e.g., traffic forecasting) where nodes have numerical features and the predicted output is a numerical value. Most of ...
0
votes
1
answer
79
views
Dummy variable trap in neural networks and class visualization
Let's say I have data records looking like that: (x1, x2, x3, x4, ..., x100), where each x can be either ...
- 101
0
votes
0
answers
56
views
saliency vs. sensitivity: proper definition and distinction
My understanding (to be critisized potentially)
In general, what I understand from saliency and sensitivity in a classification problem is:
Sensitivity means how sensitive is the predicted class to ...
- 395
0
votes
1
answer
102
views
What exactly is the AI explainability problem?
I am pretty new to AI and have recently been paying attention to AI explainability and the fact that it remains a hurdle within the path of commercializing certain AI systems in health for instance. I ...
- 1
0
votes
1
answer
249
views
Why gradients are used in Layer-wise Relevance Propagation (LRP)?
To give you an overview, Layer-wise Relevance Propagation is a technique by which we can get relevance values at each node of the neural network. These calculated relevance values (per node) are ...
- 111
0
votes
1
answer
87
views
Is the Machine Learning community going against Occam's razor?
I have been using ML models, for a couple of years, but I am actually in the neuroscience field. In it, mathematical models try to assume the smaller number of things and make hypothesis as simple as ...
- 11
1
vote
0
answers
40
views
Examples of self-explainable models used in NLP other than prototype-based
I am looking for all the methods used in NLP that are self-explainable, or explainable by design. That is to say, the ones that use the predictive model itself to explain the entire model's predictive ...
0
votes
0
answers
55
views
What are some solid metrics to evaluate/compare the outputs of explainable algorithms?
Consider a learned CNN image classifier and a task that focuses on studying the outputs of explainable algorithms, such as integrated gradients and grad-cam, on the classifier's predictions.
I am ...
- 113
0
votes
0
answers
17
views
What are confounding information, things that could also explain the outcome in a performance attribution model?
I am reading a paper about a performance attribution method, a method for finding the degree to which an outcome can be attributed to parts of a text while controlling for potential confounders. I ...
0
votes
0
answers
21
views
How to interpret integrated gradients in an NLP toxic text classification use-case?
I am trying to understand how integrated gradients work in the NLP case.
Let $F: \mathbb{R}^{n} \rightarrow[0,1]$ a function representing a neural network, $x \in \mathbb{R}^{n}$ an input and $x' \in ...
0
votes
1
answer
535
views
What is the state of the art concerning autoencoder that connect 2 images that are not similar but are physicaly related?
I am currently working on an autoencoder that connect two images. The first one can be seen as the electron flow and the second one is the electrostatic potential seen by the electrons. Long story ...
2
votes
1
answer
130
views
Can AlphaZero develop significantly different playing styles (depending on the random games from which it learrns)?
There is a quite popular video analysing a chess game AlphaZero vs. AlphaZero, called "the perfect game". It leaves some questions open and I'd like to ask them here:
Did the two copies of ...
1
vote
0
answers
63
views
Examples of rationalizable AI
The marvelous book Game Changer: AlphaZero's Groundbreaking Chess Strategies and the Promise of AI gave rise to this question. It is - in my opinion - a perfect example of rationalizing a piece of AI ...
0
votes
2
answers
459
views
How to count overlapping objects with neural networks
Consider the following task to be solved by a neural network: Given a $N\times N$ pixel grid with up to $M$ objects drawn on it, either squares (9 pixels) or diamonds (5 pixels):
square
diamond
The ...
1
vote
1
answer
68
views
What does the complexity equation constitute exactly in “Why Should I Trust You?” LIME paper
I've recently been reading this paper on LIME, an algorithm to interpret ANY machine learning model. I encountered this equation (in red) on page 4 and have just been having a hard time deciphering ...
1
vote
0
answers
184
views
What exactly do gradient-based saliency map tell us?
As far as I understand, gradients are supposed to tell us 1) the magnitude and 2) direction, to update a parameter such as to minimize the loss function.
Regarding saliency maps, which use gradients ...
- 21
0
votes
0
answers
47
views
Explainable AI for complex input features
I have a model for binary classification that includes 2 linear layers with RELU activation function and Sigmoid in the last layer. The input features are FastText word embedding, frequency, and ...
4
votes
1
answer
103
views
How can I interpret the way the neural network is producing an output for a given input?
I'm using a small neural network (2 hidden layers, 60 neurons apiece) for a rather complex binary classification problem.
The network works well, but I'd like to know how it is using the inputs to ...
- 153
1
vote
0
answers
20
views
What does the lambda parameter in the paper "Interpretable Explanations of Black Boxes by Meaningful Perturbation" do?
I do not understand the purpose of the $\lambda$ parameter in equation 3 of the paper Interpretable Explanations of Black Boxes by Meaningful Perturbation.
$$m^{*}=\underset{m \in[0,1]^{\Lambda}}{\...
- 465
4
votes
1
answer
369
views
Are these visualisations the filters of the convolution layer or the convolved images with the filters?
There are several images related to convolutional networks on the Internet, an example of which I have given below
My question is: are these images the weights/filters of the convolution layer (the ...
- 111
0
votes
1
answer
118
views
Why don't integrated gradients explain samples correctly?
I have a linear tabular dataset made of floats. The dataset follows a simple rule like:
...
0
votes
2
answers
172
views
One hot encoding vs dummy variables best practices for explainable AI (XAI)
When creating artificial columns for your categorical variables there are two mainstream methods you could use:
Disclaimer: For this example, I use the following definitions of dummy variables and one-...
- 101
1
vote
1
answer
266
views
Why are CNN binary classifier output probability distributions often skewed?
I've been working on a lot of simple resnet18 binary classifiers lately and I've started to notice that the probability distributions are often skewed one way or the other. This figure shows one such ...
- 1,259
2
votes
0
answers
38
views
What is the paper that states that humans incorrectly trust the incorrect explanations of the AI?
I was reading a paper on the subject of explainable AI and interpretability, in particular the tendency of people (even experts) to excessively trusting explanations given by AI. In the intro the ...
- 21
3
votes
1
answer
474
views
In GradCAM, why is activation strength considered an indicator of relevant regions?
In the GradCAM paper section 3 they implicitly propose that two things are needed to understand which areas of an input image contribute most to the output class (in a multi-label classification ...
- 1,259
3
votes
1
answer
162
views
What exactly is an interpretable machine learning model?
From this page in Interpretable-ml book and this article on Analytics Vidhya, it means to know what has happened inside an ML model to arrive at the result/prediction/conclusion.
In linear regression, ...
2
votes
1
answer
175
views
Black Box Explanations: Using LIME and SHAP in python
Recently, I came across the paper Robust and Stable Black Box Explanations, which discusses a nice framework for global model-agnostic explanations.
I was thinking to recreate the experiments ...
- 21
1
vote
1
answer
58
views
What do the notations $\sim$ and $\Delta (A) $ mean in the paper "Fairness Through Awareness"?
In this paper Fairness Through Awareness, the notation $\mathbb{E}_{x \sim V} \mathbb{E}_{a \sim \mu_x} L(x,a)$ is being used (page 5 top line), where $V$ denotes the set of individuals (so I guess ...
- 175
1
vote
0
answers
59
views
Is it possible to create a fair machine learning system?
I started thinking about the fairness of machine learning models recently.
Wiki page for Fairness_(machine_learning) defines fairness as:
In machine learning, a given algorithm is said to be fair, ...
- 111
2
votes
2
answers
50
views
Can a NN be configured to indicate which points of the input influenced its prediction and how?
Suppose I want to classify a dataset like the MNIST handwritten dataset, but it has added distractions. For example, here we have a 6 but with extra strokes around it that don't add value.
I suppose ...
- 1,259
6
votes
0
answers
139
views
Has anyone attempted to take a bunch of similar neural networks to extract general formulae about the focus area? [closed]
When a neural network learns something from a data set, we are left with a bunch of weights which represent some approximation of knowledge about the world. Although different data sets or even ...
- 161
3
votes
1
answer
164
views
What needs to be done to make a fair algorithm?
What needs to be done to make a fair algorithm (supervised and unsupervised)?
In this context, there is no consensus on the definition of fairness, so you can use the definition you find most ...
- 579
2
votes
3
answers
212
views
Who is working on explaining the knowledge encoded into machine learning models? [duplicate]
The thing about machine learning (ML) that worries me is that "knowledge" acquired in ML is hidden: we usually can't explain the criteria or methods used by the machine to provide an answer ...
- 363
69
votes
9
answers
11k
views
Why do we need explainable AI?
If the original purpose for developing AI was to help humans in some tasks and that purpose still holds, why should we care about its explainability? For example, in deep learning, as long as the ...
- 2,621
4
votes
1
answer
371
views
Is tabular Q-learning considered interpretable?
I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent ...
- 141
2
votes
1
answer
104
views
Is explainable AI more feasible through symbolic AI or soft computing?
Is explainable AI more feasible through symbolic AI or soft computing?
How much each paradigm, symbolic AI and soft computing (or hybrid approaches), addresses explanation and argumentation, where ...
18
votes
3
answers
3k
views
Which explainable artificial intelligence techniques are there?
Explainable artificial intelligence (XAI) is concerned with the development of techniques that can enhance the interpretability, accountability, and transparency of artificial intelligence and, in ...
- 37.1k
1
vote
0
answers
150
views
How can we create eXplainable Artificial Intelligence? [duplicate]
Currently, we can build the Artificial Intelligence (AI) approaches that respectively explain their actions within the use of goal trees 1. By moving up and down across the tree, it keeps tracking the ...
5
votes
2
answers
490
views
What do the neural network's weights represent conceptually?
I understand how neural networks work and have studied their theory well.
My question is: On the whole, is there a clear understanding of how mutation occurs within a neural network from the input ...
- 151
1
vote
0
answers
57
views
What should we do when the new data drastically change the current model?
In machine learning (in particular, supervised learning), if some new data changes the previous model/function drastically, then I think we should study that data. Does it happen? How to handle such a ...
- 338
9
votes
1
answer
396
views
Why does nobody use decision trees for visual question answering?
I'm starting a project that will involve computer vision, visual question answering, and explainability. I am currently choosing what type of algorithm to use for my classifier - a neural network or a ...
9
votes
2
answers
404
views
How is the "right to explanation" reasonable?
There has been recent uptick in interest in eXplainable Artificial Intelligence (XAI). Here is XAI's mission as stated on its DARPA page:
The Explainable AI (XAI) program aims to create a suite of ...
- 91
2
votes
3
answers
164
views
Why are tree-based models more widely used in Medical Diagnosis?
In Chapter 14.4 (p. 664) of the book Pattern Recognition and Machine Learning by Bishop, it is mentioned that tree-based models are more widely used in Medical Diagnosis.
Apart from giving better ...
- 499
9
votes
1
answer
252
views
How would one debug, understand or fix the outcome of a neural network?
It seems fairly uncontroversial to say that NN based approaches are becoming quite powerful tools in many AI areas - whether recognising and decomposing images (faces at a border, street scenes in ...
- 233
95
votes
7
answers
18k
views
Do scientists know what is happening inside artificial neural networks?
Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant? Do they understand ...
- 10.3k