Skip to main content

Questions tagged [uncertainty-quantification]

For questions about uncertainty quantification (aka uncertainty estimation) in the context of artificial intelligence, in particular, in the context of Bayesian machine learning.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Is it possible to have lower ECE but worse reliable curve?

I am new to the calibration concept for classification. I have tried temperature scaling on my model's results. However, after applying temperature scaling, the reliable curve got worse despite ...
Sara's user avatar
  • 1
0 votes
1 answer
17 views

Understanding different methods of covariance parametrization

In the paper https://arxiv.org/pdf/2112.02143 it is noted that there are different ways to "parametrize" covariance. (page 4) What does "parametrizing" covariance mean exactly? In ...
Homer Sanchez's user avatar
0 votes
0 answers
66 views

Why using a pretrained model with dense variational for uncertainty estimates gives low accuracy?

I try to a train model for real image classification and to estimate uncertainty by using the TensorFlow Probability library. For that, I am using a pretrained model as a feature extractor (the ...
a-eng's user avatar
  • 1
1 vote
1 answer
73 views

How Mutual Information is related to uncertainty

I'm studying the chapter of Information theory from Haykin's deep learning book. It says Mutual Information between two continuous random variables $X,Y$ is defined in terms of the differential ...
piero's user avatar
  • 123
1 vote
2 answers
410 views

How do language models know what they don't know - and report it?

Again and again I ask myself what goes on in a pre-trained transformer-based language model (like ChatGPT9) when it comes to "know" that it cannot give an appropriate answer and either ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
402 views

What does AUSE metric mean in uncertainty estimation

I am reading the paper "Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision", I do not understand the definition of AUSE metric in this sentence "but only in ...
TimothyShi's user avatar
1 vote
0 answers
202 views

Active Learning regression with Random Forest

I have a dataset of about 8k points and I am trying to employ active learning with the random forest regressor. I have split the dataset to train and ...
ado sar's user avatar
  • 150
1 vote
0 answers
72 views

How to calculate uncertainty in Deep Ensembles for Reinforcement Learning?

Lets take the following example: I must predict the return (Q-values) of x state-action pairs using an ensemble of m models. Using NumPy I could have the following for x = 5 and m = 3: ...
HenDoNR's user avatar
  • 81
0 votes
1 answer
99 views

Why would the "improvement" be the result of random initialization, and so why should we use multiple runs?

I got this feedback for my thesis paper. The improvement shown in the results section could be the result of random initialization. There should be multiple runs with means and standard deviations. ...
Md. Asif Iqbal Fahim's user avatar
1 vote
0 answers
35 views

Does MobileNet SSD v2 only capture aleatoric uncertainty (and so not the epistemic one)?

Regarding the MobileNet SSD v2 model, I was wondering to what extend it captures uncertainty of the predictions. There are 2 types of uncertainty, data uncertainty (aleatoric) and model uncertainty (...
Baka's user avatar
  • 11
0 votes
0 answers
57 views

What are the standard ways to measure the quality of a set of numerical predictions that include uncertainties?

I have a radial basis function that supplies uncertainties (standard deviations) with its predictions, which are numerical values. This function is computed for a particular point by computing its ...
PJ7's user avatar
  • 101
3 votes
0 answers
60 views

Do we need as much information to know if we can can answer a question as we need to actually answer the question?

I am reading The Book of Why: The New Science of Cause and Effect by Judea Pearl, and in page 12 I see the following diagram. The box on the right side of box 5 "Can the query be answered?" ...
Lerner Zhang's user avatar
3 votes
0 answers
88 views

Why does this formula $\sigma^2 + \frac{1}{T}\sum_{t=1}^Tf^{\hat{W_t}}(x)^Tf^{\hat{W_t}}(x_t)-E(y)^TE(y)$ approximate the variance?

How does: $$\text{Var}(y) \approx \sigma^2 + \frac{1}{T}\sum_{t=1}^Tf^{\hat{W_t}}(x)^Tf^{\hat{W_t}}(x_t)-E(y)^TE(y)$$ approximate variance? I'm currently reading What Uncertainties Do We Need in ...
user8714896's user avatar
3 votes
0 answers
499 views

How can I use Monte Carlo Dropout in a pre-trained CNN model?

In Monte Carlo Dropout (MCD), I know that I should enable dropout during training and testing, then get multiple predictions for the same input $x$ by performing multiple forward passes with $x$, then,...
lebebop's user avatar
  • 31
1 vote
2 answers
1k views

Why is my Keras prediction always close to 100% for one image class?

I am using Keras (on top of TF 2.3) to train an image classifier. In some cases I have more than two classes, but often there are just two classes (either "good" or "bad"). I am ...
Matthias's user avatar
  • 165
4 votes
1 answer
126 views

Is there any research on models that provide uncertainty estimation?

Is there any research on machine learning models that provide uncertainty estimation? If I train a denoising autoencoder on words and put through a noised word, I'd like it to return a certainty that ...
user8714896's user avatar
37 votes
6 answers
11k views

Why do CNN's sometimes make highly confident mistakes, and how can one combat this problem?

I trained a simple CNN on the MNIST database of handwritten digits to 99% accuracy. I'm feeding in a bunch of handwritten digits, and non-digits from a document. I want the CNN to report errors, so I ...
Alexander Soare's user avatar
7 votes
1 answer
3k views

How does the Dempster-Shafer theory differ from Bayesian reasoning?

How does the Dempster-Shafer theory differ from Bayesian reasoning? How do these two methods handle uncertainty and compute posterior distributions?
rudresh dwivedi's user avatar
5 votes
5 answers
448 views

How would AI be able to self-examine?

As I see some cases of machine-learning based artificial intelligence, I often see they make critical mistakes when they face inexperienced situations. In our case, when we encounter totally new ...
A Cat Named Tiger's user avatar