Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [explainable-ai]

For questions related to explainable artificial intelligence (XAI), also known as interpretable AI, which refers to AI techniques that can be trusted and easily understood by humans, which are particularly relevant in areas like healthcare or self-driving cars. There are several concepts related to XAI, such as accountability, fairness, and transparency.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Do I need model calibration/feature normalization for reliable path integrated gradient/sampled Shapley feature attribution in a dnn model?

Are model calibration and feature normalization required for path integrated gradient and sampled Shapley-based feature attribution analyses to work properly in a deep neural network model? I read ...
AiWannabe's user avatar
0 votes
0 answers
14 views

Why in Vanilla Gradient for saliency map, we set other classes to zero?

I am reading Pixel Attribution (Saliency Maps) and I have stumbled on the following. For the Vanilla Gradient, if we want to calculate the saliency map for image $I$, then we start with the following ...
ado sar's user avatar
  • 150
0 votes
0 answers
25 views

Calculating class-specific permutation feature importances for multilabel classification problem

I would like to apply the permutation feature importance technique to rank the features of a siamese network model that I trained. I am currently using this siamese network to perform some kind of ...
ashenflower's user avatar
0 votes
1 answer
50 views

how can I interpret attention weights matrix? Are they reliable?

I've fine-tuned two different models (Bert and Roberta) on a dataset for a binary classification task and I'm comparing the sentences where the models predict wrong. I decided to use attention weights ...
Shayan's user avatar
  • 21
1 vote
2 answers
236 views

Which techniques are best suitable for explainable AI for LLM models

I am currently working with large language models like llama and mistral, interested in techniques for making these models more explainable. I am looking for some tools or techniques which can help me ...
Hiren Namera's user avatar
1 vote
0 answers
43 views

Integrated gradients on text to text models

I am trying to apply integrated gradients (using library captum) on a text generation model. Specifically, it is a model that generates patches for input buggy code. I want to know if applying the ...
Nimantha's user avatar
  • 111
1 vote
0 answers
202 views

What is 'system card'?

What is 'system card' in these context: https://ai.meta.com/blog/system-cards-a-new-resource-for-understanding-how-ai-systems-work/ Additionally, individual model developers may provide ...
Vy Do's user avatar
  • 111
0 votes
1 answer
69 views

What is Explainable AI and what does it strive for?

I understand the need for Explainability in AI. However, I am uncertain of what is meant by 'making AI explainable'. What needs to be explainable? Is it the output of a model? Does it refer to the ...
Robin van Hoorn's user avatar
0 votes
0 answers
20 views

Can we use bounding box cropping to avoid shortcut learning (achieve explainable AI)?

Deep neural networks sometime use shortcut features (pseudo correlation) to predict. For example, in cat-dog classification, the network may use the background information (e.g. floor, grass) as a ...
Mingzhou Liu's user avatar
0 votes
0 answers
211 views

Feature Importance for FT Transformer

I hope someone is able to shed light on this. I was reading through the codes on this link with regards to getting feature importance from attention scores. https://github.com/aruberts/...
DDM's user avatar
  • 101
1 vote
2 answers
380 views

How do language models know what they don't know - and report it?

Again and again I ask myself what goes on in a pre-trained transformer-based language model (like ChatGPT9) when it comes to "know" that it cannot give an appropriate answer and either ...
Hans-Peter Stricker's user avatar
0 votes
2 answers
585 views

Why shouldn't the attention matrices $W^Q$, $W^K$, $W^V$ be the same?

My question is why the attention head matrices $W^Q$, $W^K$, $W^V$ should not be the same $W = W^Q =W^K= W^V$. In my understanding of transformer-based language models one attention head is ...
Hans-Peter Stricker's user avatar
2 votes
1 answer
813 views

Surveys/Important papers in Explainability for LLM?

I'm interested in the topic of Explainability for LLM: the attempts to find some higher understandable structures inside the LLMs or, to put it simply (though may be not completely correctly), the ...
lithuak's user avatar
  • 121
1 vote
0 answers
101 views

In terms of explainability, is attentive RNN easier to explain than the transformer?

Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to ...
hH1sG0n3's user avatar
  • 211
2 votes
1 answer
60 views

Is there any interpretation method suitable for CNNs which do regression tasks?

I mainly tackle regression problems by CNNs, and want to find a reliable method to calculate the heatmaps for NN's results. However, I find almost all interpretation methods including CAM is used for ...
minghuisvn's user avatar
1 vote
1 answer
830 views

Are there Explainable GNN methods for node regression tasks?

I am wondering if there are gnn explainable methods for a regression task (e.g., traffic forecasting) where nodes have numerical features and the predicted output is a numerical value. Most of ...
Achiles Br's user avatar
0 votes
1 answer
205 views

Dummy variable trap in neural networks and class visualization

Let's say I have data records looking like that: (x1, x2, x3, x4, ..., x100), where each x can be either ...
leleogere's user avatar
  • 101
0 votes
1 answer
192 views

What exactly is the AI explainability problem?

I am pretty new to AI and have recently been paying attention to AI explainability and the fact that it remains a hurdle within the path of commercializing certain AI systems in health for instance. I ...
rp2001's user avatar
  • 1
1 vote
1 answer
708 views

Why gradients are used in Layer-wise Relevance Propagation (LRP)?

To give you an overview, Layer-wise Relevance Propagation is a technique by which we can get relevance values at each node of the neural network. These calculated relevance values (per node) are ...
Ritwik's user avatar
  • 121
0 votes
1 answer
162 views

Is the Machine Learning community going against Occam's razor?

I have been using ML models, for a couple of years, but I am actually in the neuroscience field. In it, mathematical models try to assume the smaller number of things and make hypothesis as simple as ...
JFR's user avatar
  • 11
1 vote
0 answers
48 views

Examples of self-explainable models used in NLP other than prototype-based

I am looking for all the methods used in NLP that are self-explainable, or explainable by design. That is to say, the ones that use the predictive model itself to explain the entire model's predictive ...
Revolucion for Monica's user avatar
1 vote
1 answer
816 views

What is the state of the art concerning autoencoder that connect 2 images that are not similar but are physicaly related?

I am currently working on an autoencoder that connect two images. The first one can be seen as the electron flow and the second one is the electrostatic potential seen by the electrons. Long story ...
Gaetan Percebois's user avatar
2 votes
1 answer
178 views

Can AlphaZero develop significantly different playing styles (depending on the random games from which it learrns)?

There is a quite popular video analysing a chess game AlphaZero vs. AlphaZero, called "the perfect game". It leaves some questions open and I'd like to ask them here: Did the two copies of ...
Hans-Peter Stricker's user avatar
1 vote
0 answers
84 views

Examples of rationalizable AI

The marvelous book Game Changer: AlphaZero's Groundbreaking Chess Strategies and the Promise of AI gave rise to this question. It is - in my opinion - a perfect example of rationalizing a piece of AI ...
Hans-Peter Stricker's user avatar
0 votes
2 answers
669 views

How to count overlapping objects with neural networks

Consider the following task to be solved by a neural network: Given a $N\times N$ pixel grid with up to $M$ objects drawn on it, either squares (9 pixels) or diamonds (5 pixels): square    diamond The ...
Hans-Peter Stricker's user avatar
1 vote
1 answer
85 views

What does the complexity equation constitute exactly in “Why Should I Trust You?” LIME paper

I've recently been reading this paper on LIME, an algorithm to interpret ANY machine learning model. I encountered this equation (in red) on page 4 and have just been having a hard time deciphering ...
loose the fools juice's user avatar
1 vote
0 answers
451 views

What exactly do gradient-based saliency map tell us?

As far as I understand, gradients are supposed to tell us 1) the magnitude and 2) direction, to update a parameter such as to minimize the loss function. Regarding saliency maps, which use gradients ...
user452306's user avatar
0 votes
0 answers
64 views

Explainable AI for complex input features

I have a model for binary classification that includes 2 linear layers with RELU activation function and Sigmoid in the last layer. The input features are FastText word embedding, frequency, and ...
Abolfazl Mohajeri's user avatar
4 votes
1 answer
144 views

How can I interpret the way the neural network is producing an output for a given input?

I'm using a small neural network (2 hidden layers, 60 neurons apiece) for a rather complex binary classification problem. The network works well, but I'd like to know how it is using the inputs to ...
asheets's user avatar
  • 153
1 vote
0 answers
22 views

What does the lambda parameter in the paper "Interpretable Explanations of Black Boxes by Meaningful Perturbation" do?

I do not understand the purpose of the $\lambda$ parameter in equation 3 of the paper Interpretable Explanations of Black Boxes by Meaningful Perturbation. $$m^{*}=\underset{m \in[0,1]^{\Lambda}}{\...
Bert Gayus's user avatar
5 votes
1 answer
560 views

Are these visualisations the filters of the convolution layer or the convolved images with the filters?

There are several images related to convolutional networks on the Internet, an example of which I have given below My question is: are these images the weights/filters of the convolution layer (the ...
Ali.kavari76's user avatar
0 votes
1 answer
178 views

Why don't integrated gradients explain samples correctly?

I have a linear tabular dataset made of floats. The dataset follows a simple rule like: ...
user46784's user avatar
0 votes
2 answers
260 views

One hot encoding vs dummy variables best practices for explainable AI (XAI)

When creating artificial columns for your categorical variables there are two mainstream methods you could use: Disclaimer: For this example, I use the following definitions of dummy variables and one-...
Koen's user avatar
  • 101
1 vote
1 answer
399 views

Why are CNN binary classifier output probability distributions often skewed?

I've been working on a lot of simple resnet18 binary classifiers lately and I've started to notice that the probability distributions are often skewed one way or the other. This figure shows one such ...
Alexander Soare's user avatar
2 votes
0 answers
41 views

What is the paper that states that humans incorrectly trust the incorrect explanations of the AI?

I was reading a paper on the subject of explainable AI and interpretability, in particular the tendency of people (even experts) to excessively trusting explanations given by AI. In the intro the ...
Stoaty193's user avatar
3 votes
1 answer
671 views

In GradCAM, why is activation strength considered an indicator of relevant regions?

In the GradCAM paper section 3 they implicitly propose that two things are needed to understand which areas of an input image contribute most to the output class (in a multi-label classification ...
Alexander Soare's user avatar
3 votes
1 answer
183 views

What exactly is an interpretable machine learning model?

From this page in Interpretable-ml book and this article on Analytics Vidhya, it means to know what has happened inside an ML model to arrive at the result/prediction/conclusion. In linear regression, ...
Naveen Reddy Marthala's user avatar
2 votes
1 answer
205 views

Black Box Explanations: Using LIME and SHAP in python

Recently, I came across the paper Robust and Stable Black Box Explanations, which discusses a nice framework for global model-agnostic explanations. I was thinking to recreate the experiments ...
user294142's user avatar
1 vote
1 answer
65 views

What do the notations $\sim$ and $\Delta (A) $ mean in the paper "Fairness Through Awareness"?

In this paper Fairness Through Awareness, the notation $\mathbb{E}_{x \sim V} \mathbb{E}_{a \sim \mu_x} L(x,a)$ is being used (page 5 top line), where $V$ denotes the set of individuals (so I guess ...
MMM's user avatar
  • 185
1 vote
0 answers
67 views

Is it possible to create a fair machine learning system?

I started thinking about the fairness of machine learning models recently. Wiki page for Fairness_(machine_learning) defines fairness as: In machine learning, a given algorithm is said to be fair, ...
matcheek's user avatar
  • 111
2 votes
2 answers
51 views

Can a NN be configured to indicate which points of the input influenced its prediction and how?

Suppose I want to classify a dataset like the MNIST handwritten dataset, but it has added distractions. For example, here we have a 6 but with extra strokes around it that don't add value. I suppose ...
Alexander Soare's user avatar
6 votes
0 answers
142 views

Has anyone attempted to take a bunch of similar neural networks to extract general formulae about the focus area? [closed]

When a neural network learns something from a data set, we are left with a bunch of weights which represent some approximation of knowledge about the world. Although different data sets or even ...
Lawnmower Man's user avatar
3 votes
1 answer
194 views

What needs to be done to make a fair algorithm?

What needs to be done to make a fair algorithm (supervised and unsupervised)? In this context, there is no consensus on the definition of fairness, so you can use the definition you find most ...
jennifer ruurs's user avatar
2 votes
3 answers
225 views

Who is working on explaining the knowledge encoded into machine learning models? [duplicate]

The thing about machine learning (ML) that worries me is that "knowledge" acquired in ML is hidden: we usually can't explain the criteria or methods used by the machine to provide an answer ...
S. McGrew's user avatar
  • 363
72 votes
9 answers
11k views

Why do we need explainable AI?

If the original purpose for developing AI was to help humans in some tasks and that purpose still holds, why should we care about its explainability? For example, in deep learning, as long as the ...
malioboro's user avatar
  • 2,819
4 votes
1 answer
487 views

Is tabular Q-learning considered interpretable?

I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent ...
harshal.c's user avatar
  • 141
2 votes
1 answer
130 views

Is explainable AI more feasible through symbolic AI or soft computing?

Is explainable AI more feasible through symbolic AI or soft computing? How much each paradigm, symbolic AI and soft computing (or hybrid approaches), addresses explanation and argumentation, where ...
StalkingAletheia's user avatar
18 votes
3 answers
3k views

Which explainable artificial intelligence techniques are there?

Explainable artificial intelligence (XAI) is concerned with the development of techniques that can enhance the interpretability, accountability, and transparency of artificial intelligence and, in ...
nbro's user avatar
  • 40.8k
1 vote
0 answers
152 views

How can we create eXplainable Artificial Intelligence? [duplicate]

Currently, we can build the Artificial Intelligence (AI) approaches that respectively explain their actions within the use of goal trees 1. By moving up and down across the tree, it keeps tracking the ...
Francisco Maria Calisto's user avatar
5 votes
2 answers
669 views

What do the neural network's weights represent conceptually?

I understand how neural networks work and have studied their theory well. My question is: On the whole, is there a clear understanding of how mutation occurs within a neural network from the input ...
user248884's user avatar