How to make machine learning model that reports ambiguity of the input?

Question

Suppose I want to build a neural network regression model that takes one input and return one output.

Here's the training data:

0.1 => 0.1
0.2 => 0.2
0.1 => -0.1

You will see that there are 2 inputs 0.1 that matches to different output values 0.1 and -0.1. So what will happen with most machine learning models is that they will predict the average when 0.1 is fed to the model. E.g. the output of 0.1 will be (0.1 + (-0.1))/2 = 0.

But this 0 as an average answer is an incorrect answer. I want the model to be telling me that the input is ambiguous/insufficient to infer the output. Ideally, the model would report it as a form of confidence.

How do I report predictability confidence from the input?

The application that I find very useful in many areas is that I could then later ask the model to show me inputs that are easy to predict and inputs that are ambiguous. This would make me able to collect the data that are making sense.

One way I know is to train the model then check the error on each training data, if it's high then it probably means that the input is ambiguous. But if you know any other papers or better techniques, I would be appreciated to know that!

Answer 1

Another specific way to do this if one uses a neural network for this. Use a dropout a layer in your network and instead of scaling the activations at test time, one can sample the activations (just like in training-time) and predict multiple times for a given input, then look at distribution of your outputs. Intuitively this would add "probabilistic, bayesian effect" to you neural network. I think this method was first proposed in Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning which is called Monte Carlo Dropout.

Answer 2

Predicting with confidence: the best machine learning idea you never heard of by Scott Locklin might provide you an idea.

The name of this basket of ideas is “conformal prediction.”