What you are suggesting is similar to active learning and reward modelling.
To summarize both quickly, active learning is used when data are scarce or when the labeling process is too time consuming (almost always the case in NLP). To speed up the process, the idea is to train a model in performing not only a task, but also an estimation of its uncertainty when performing that task. For example, the model could output a classification and a probability associated with that classification that tells how much the model is 'unsure' of that classification. These probabilities can then be leveraged to collect a bunch of out of train data where the model performs poorly (i.e. low probability scores). These data are then annotated by a human, and used to expand the initial dataset and retrain the model. The whole process is meant to minimize the amount of annotation while maximizing the performance of the model. Some people use active learning also on large datasets, the idea being that many training instances are similar to each other and redundant for the model, so by focusing on training the model in instances that are particularly difficult for the model to capture, the training time can be reduce by a large margin.
Reward modelling is more interesting, and it is relate to reinforcement learning. Unlike in classic deep learning, which focus mainly in the design of loss functions, in reinforcement learning the main component is the reward function. A reward function is much trickier to design compare to a loss function, cause it's usually impossible to predict all possible non useful strategies that an agent might learn (e.g. running in circles to prevent to hit obstacles). So, to compensate for this difficulties, some people came up with the idea of letting artificial agents to design their own reward function, with the only constrain of an external human annotator penalizing dumb functions that emerge during the process. It's a bit hard to explain the idea shortly, but I personally find it really fascinating, and I suggest you also to take a look at this video for a more complete explanation.