Using a pre-trained model to generate labels to data to then train a model on

Question

I'm trying to set up a pipeline for my ML models to automatically re-train themselves whenever concept drift occurs to recalibrate to the new output distributions. However, I can't get ground-truth from my data without manual labeling, and I want an automated MLOps pipeline, so that no human-in-the-loop labeling is required.

I was noticing that using zero-shot learning using BART was actually giving pretty impressive results with my label set on sample data I want to classify on.

Suppose I were to use BART to manually label data, say, if it scored my label with a score of 0.95 or better, and then just used that labeled data to train my models.

Is this "cheating"? Am I trying to cut a corner here that can't be cut? Will this create a meaningful model for my text data, or just cause my trained model to try to become a crude version of the BART model that provided the labels in the first place, and therefore just be worse than strictly using the BART model?

Answer 1

If using BART is already giving you good results, why do you need a new model?

Not a rhetorical question. You might have good reasons for that. Training a model with less parameters optimized only on few classes is an example. But if you can't came up with good reasons then the answer is pretty straight forward: just use BART. Reason being that a model trained on another model annotations without any human in the loop will never perform better than the model used to automatically train the data. The first reason being error propagation, like in any other area of empirical science. And on a more technical level you can't really use tricks like using only predictions with a high confidence scores, cause deep learning models are basically never calibrated, hence you can't trust them and a human would still have to check them manually.