Using Human Confirmation in place of a loss Function for Training

Question

Has there been any experimentation in designing an AI to prompt a human to judge the accuracy of it's outcomes? instead of using a loss function, a human can judge the accuracy of it's estimation using some kind of metric, where it can then use that too update it's weights.

I was looking for some feedback on whether this is a plausible idea.

I was thinking that for domains that lack sufficient training data to solve problems this could be a possible solution.

Of course, it isn't feasible to judge every iteration of a training loop. So maybe feedback could be provided for the average of a number of estimations. Maybe every 100 estimations you could provide feedback.

It may not be a great training method because of the sparsity of feedback, but it could provide a place to start if you don't have a lot of data to throw at your problem initially.

Answer 1

What you are suggesting is similar to active learning and reward modelling.

To summarize both quickly, active learning is used when data are scarce or when the labeling process is too time consuming (almost always the case in NLP). To speed up the process, the idea is to train a model in performing not only a task, but also an estimation of its uncertainty when performing that task. For example, the model could output a classification and a probability associated with that classification that tells how much the model is 'unsure' of that classification. These probabilities can then be leveraged to collect a bunch of out of train data where the model performs poorly (i.e. low probability scores). These data are then annotated by a human, and used to expand the initial dataset and retrain the model. The whole process is meant to minimize the amount of annotation while maximizing the performance of the model. Some people use active learning also on large datasets, the idea being that many training instances are similar to each other and redundant for the model, so by focusing on training the model in instances that are particularly difficult for the model to capture, the training time can be reduce by a large margin.

Reward modelling is more interesting, and it is relate to reinforcement learning. Unlike in classic deep learning, which focus mainly in the design of loss functions, in reinforcement learning the main component is the reward function. A reward function is much trickier to design compare to a loss function, cause it's usually impossible to predict all possible non useful strategies that an agent might learn (e.g. running in circles to prevent to hit obstacles). So, to compensate for this difficulties, some people came up with the idea of letting artificial agents to design their own reward function, with the only constrain of an external human annotator penalizing dumb functions that emerge during the process. It's a bit hard to explain the idea shortly, but I personally find it really fascinating, and I suggest you also to take a look at this video for a more complete explanation.