Suppose I want to build a neural network regression model that takes one input and return one output.
Here's the training data:
0.1 => 0.1 0.2 => 0.2 0.1 => -0.1
You will see that there are 2 inputs
0.1 that matches to different output values
-0.1. So what will happen with most machine learning models is that they will predict the average when
0.1 is fed to the model. E.g. the output of
0.1 will be
(0.1 + (-0.1))/2 = 0.
0 as an average answer is an incorrect answer. I want the model to be telling me that the input is ambiguous/insufficient to infer the output. Ideally, the model would report it as a form of confidence.
How do I report predictability confidence from the input?
The application that I find very useful in many areas is that I could then later ask the model to show me inputs that are easy to predict and inputs that are ambiguous. This would make me able to collect the data that are making sense.
One way I know is to train the model then check the error on each training data, if it's high then it probably means that the input is ambiguous. But if you know any other papers or better techniques, I would be appreciated to know that!