Questions tagged [explainable-ai]

For questions related to explainable artificial intelligence (XAI), also known as interpretable AI, which refers to AI techniques that can be trusted and easily understood by humans, which are particularly relevant in areas like healthcare or self-driving cars. There are several concepts related to XAI, such as accountability, fairness, and transparency.

Filter by
Sorted by
Tagged with
90
votes
7answers
16k views

Do scientists know what is happening inside artificial neural networks?

Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant? Do they understand ...
68
votes
9answers
10k views

Why do we need explainable AI?

If the original purpose for developing AI was to help humans in some tasks and that purpose still holds, why should we care about its explainability? For example, in deep learning, as long as the ...
13
votes
3answers
2k views

Which explainable artificial intelligence techniques are there?

Explainable artificial intelligence (XAI) is concerned with the development of techniques that can enhance the interpretability, accountability, and transparency of artificial intelligence and, in ...
9
votes
2answers
385 views

How is the "right to explanation" reasonable?

There has been recent uptick in interest in eXplainable Artificial Intelligence (XAI). Here is XAI's mission as stated on its DARPA page: The Explainable AI (XAI) program aims to create a suite of ...
8
votes
1answer
229 views

How would one debug, understand or fix the outcome of a neural network?

It seems fairly uncontroversial to say that NN based approaches are becoming quite powerful tools in many AI areas - whether recognising and decomposing images (faces at a border, street scenes in ...
6
votes
1answer
228 views

Why does nobody use decision trees for visual question answering?

I'm starting a project that will involve computer vision, visual question answering, and explainability. I am currently choosing what type of algorithm to use for my classifier - a neural network or a ...
6
votes
0answers
134 views

Has anyone attempted to take a bunch of similar neural networks to extract general formulae about the focus area? [closed]

When a neural network learns something from a data set, we are left with a bunch of weights which represent some approximation of knowledge about the world. Although different data sets or even ...
4
votes
1answer
203 views

Is tabular Q-learning considered interpretable?

I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent ...
3
votes
1answer
118 views

Are these visualisations the filters of the convolution layer or the convolved images with the filters?

There are several images related to convolutional networks on the Internet, an example of which I have given below My question is: are these images the weights/filters of the convolution layer (the ...
3
votes
1answer
117 views

What exactly is an interpretable machine learning model?

From this page in Interpretable-ml book and this article on Analytics Vidhya, it means to know what has happened inside an ML model to arrive at the result/prediction/conclusion. In linear regression, ...
3
votes
1answer
126 views

What needs to be done to make a fair algorithm?

What needs to be done to make a fair algorithm (supervised and unsupervised)? In this context, there is no consensus on the definition of fairness, so you can use the definition you find most ...
3
votes
1answer
49 views

How can I interpret the way the neural network is producing an output for a given input?

I'm using a small neural network (2 hidden layers, 60 neurons apiece) for a rather complex binary classification problem. The network works well, but I'd like to know how it is using the inputs to ...
3
votes
1answer
207 views

In GradCAM, why is activation strength considered an indicator of relevant regions?

In the GradCAM paper section 3 they implicitly propose that two things are needed to understand which areas of an input image contribute most to the output class (in a multi-label classification ...
3
votes
1answer
122 views

Black Box Explanations: Using LIME and SHAP in python

Recently, I came across the paper Robust and Stable Black Box Explanations, which discusses a nice framework for global model-agnostic explanations. I was thinking to recreate the experiments ...
2
votes
2answers
45 views

Can a NN be configured to indicate which points of the input influenced its prediction and how?

Suppose I want to classify a dataset like the MNIST handwritten dataset, but it has added distractions. For example, here we have a 6 but with extra strokes around it that don't add value. I suppose ...
2
votes
1answer
83 views

Is explainable AI more feasible through symbolic AI or soft computing?

Is explainable AI more feasible through symbolic AI or soft computing? How much each paradigm, symbolic AI and soft computing (or hydrid approaches), adresses explanation and argumentation, where ...
2
votes
3answers
136 views

Why are tree-based models more widely used in Medical Diagnosis?

In Chapter 14.4 (p. 664) of the book Pattern Recognition and Machine Learning by Bishop, it is mentioned that tree-based models are more widely used in Medical Diagnosis. Apart from giving better ...
2
votes
0answers
38 views

What is the paper that states that humans incorrectly trust the incorrect explanations of the AI?

I was reading a paper on the subject of explainable AI and interpretability, in particular the tendency of people (even experts) to excessively trusting explanations given by AI. In the intro the ...
2
votes
3answers
195 views

Who is working on explaining the knowledge encoded into machine learning models? [duplicate]

The thing about machine learning (ML) that worries me is that "knowledge" acquired in ML is hidden: we usually can't explain the criteria or methods used by the machine to provide an answer ...
1
vote
1answer
49 views

What do the notations $\sim$ and $\Delta (A) $ mean in the paper "Fairness Through Awareness"?

In this paper Fairness Through Awareness, the notation $\mathbb{E}_{x \sim V} \mathbb{E}_{a \sim \mu_x} L(x,a)$ is being used (page 5 top line), where $V$ denotes the set of individuals (so I guess ...
1
vote
1answer
37 views

Why are CNN binary classifier output probability distributions often skewed?

I've been working on a lot of simple resnet18 binary classifiers lately and I've started to notice that the probability distributions are often skewed one way or the other. This figure shows one such ...
1
vote
0answers
27 views

Explainable AI for complex input features

I have a model for binary classification that includes 2 linear layers with RELU activation function and Sigmoid in the last layer. The input features are FastText word embedding, frequency, and ...
1
vote
0answers
14 views

What does the lambda parameter in the paper "Interpretable Explanations of Black Boxes by Meaningful Perturbation" do?

I do not understand the purpose of the $\lambda$ parameter in equation 3 of the paper Interpretable Explanations of Black Boxes by Meaningful Perturbation. $$m^{*}=\underset{m \in[0,1]^{\Lambda}}{\...
1
vote
0answers
12 views

Why can we compute mutual information in deep neural networks in information bottleneck context?

In the famous Information bottleneck paper by Tishby(https://arxiv.org/abs/1703.00810), the author proposed a framework that the neural network can compress information. And they computed the mutual ...
1
vote
0answers
207 views

What could be a good way to visualise the feature extraction process with MobileNet?

I am trying to create a visualisation for how transfer learning (feature extraction in particular) works with MobileNet. With the ml5.js library, you can extract a ...
1
vote
0answers
56 views

Is it possible to create a fair machine learning system?

I started thinking about the fairness of machine learning models recently. Wiki page for Fairness_(machine_learning) defines fairness as: In machine learning, a given algorithm is said to be fair, ...
1
vote
0answers
137 views

How can we create eXplainable Artificial Intelligence? [duplicate]

Currently, we can build the Artificial Intelligence (AI) approaches that respectively explain their actions within the use of goal trees 1. By moving up and down across the tree, it keeps tracking the ...
1
vote
0answers
54 views

What should we do when the new data drastically change the current model?

In machine learning (in particular, supervised learning), if some new data changes the previous model/function drastically, then I think we should study that data. Does it happen? How to handle such a ...
0
votes
1answer
61 views

Why don't integrated gradients explain samples correctly?

I have a linear tabular dataset made of floats. The dataset follows a simple rule like: ...
0
votes
2answers
91 views

One hot encoding vs dummy variables best practices for explainable AI (XAI)

When creating artificial columns for your categorical variables there are two mainstream methods you could use: Disclaimer: For this example, I use the following definitions of dummy variables and one-...
0
votes
0answers
72 views

How to compare multiple one-class variational autoencoders?

I have trained multiple one-class vanilla variational autoencoders that each learn the distribution of one class and have the same architecture. The classes are mostly discrete, but there are several ...
0
votes
0answers
48 views

Explainable AI for Model Comparison

In the last years a lot of different Explainable AI solutions were released. I have been looking for a solution, which is able to compare different models in real use cases. For example, you have a ...