Suppose I want to predict cats and dogs, but with a twist: the model can choose the image to predict.
For example: Given a list of 10 images (with both dogs and cats), the model need to choose one image that it's most confident of. And I'm going to evaluate its accuracy based on the image it chose only.
If I have to name this problem, I'm going to name it Learning-to-Choose (LTC). One of the easiest way to solve this problem is to train the model to predict every image independently like usual. And when it's given 10 images, just choose the image that it predicts with the highest confidence and evaluate accuracy based on that. But I don't think this is the best method because the training process didn't care about choosing anything at all.
One of my other idea is to treat the 10 images as one sample and modify the loss function to care only the one image that the model predicts with the highest confidence. But I'm concerned about training stability because the image with the highest confidence is going to change on every training step.
I am aware of the well-known problem called Learning-to-Rank (LTR) which is similar to my problem but it's not the same. LTR is trained based on the continuous values but I have categorical values.
Does anyone know of a machine learning technique/literature that is trying to solve a problem like this?