7
$\begingroup$

I plan to use my predictions as ground truth to continue training my model. These predictions are of course reviewed during this process. Is there an argument against that (reinforcement of slight mistakes/overfitting etc.)?


Here my specific use case described:

I am using detectron's faster R-CNN implementation to train a (pretrained) model to find defects of a machine part in images.

The goal is to find bounding boxes around these defects and to label them.

A colleague labeled some of the images (1500 images, making up 20% of the whole set) and I used those to train my model. Then I had the model predict defects on all 7500 images. My colleague asked me if he can review the predictions (and adjust/add if necessary) so he doesn't have to label the remaining images from scratch, and then I would like to continue the training with all the images.

$\endgroup$

2 Answers 2

5
$\begingroup$

Using the (unchecked) predictions of the model as training data is an approach known as "pseudo-labeling". It can help in certain situations, depending on the underlying structure of your dataset, but you have to be a bit careful about how you use it (e.g. only using high-confidence predictions as your pseudo-labels) and you always want to keep your pseudo-labels separate from your true labels, so you can potentially update them as your model changes.


But it sounds like you're not using the raw predictions as labels, but rather using the predictions of the models as a pointer to (currently unlabeled) examples which you then will manually label.

"Training on errors" is recognized mode of augmenting your training dataset, especially for "on-line" style systems where you're getting a continuous stream of new examples. The concept is to identify those examples which are predicted either inaccurately or with low confidence, identify the accurate labels for these instances, and then include them with the rest of the training set to help improve the predictions for similar sorts of examples in the future.

In contrast to pseudo-labeling, you're looking to correct the low confidence examples or the incorrectly predicted. Adding in high-confidence examples doesn't gain you much, as your current training set is already sufficient to correctly predict these. And with an on-line model where you're continually getting new examples, adding the well-predicted examples to your training set does potentially cause issues with subclass imbalance issues, as "normal" examples are expected to swamp out the rare outliers.


But it sounds like you have a fixed-size training set. In that case, the standard recommendation for the best course of action of dealing with unlabeled data applies: "pay someone to label it for you". What you're looking for is accurate labeling. How you get that is left somewhat nebulous, so long as the labeling is accurate. Using model results as a starting point is perfectly valid, assuming that whoever is doing the checking/correction is willing to actually do all the corrections (to the same quality level as a "from scratch" prediction) and won't wave through the model predictions as "ehh, good enough".

In addition to label accuracy, another issue may be selection bias (that is, the model may have certain subsets of examples which it performs worse/better on, and picking which examples to include in labeling on that basis may bias future training). But if you have a pre-determined, fixed-size training set this is not really an issue if you label all of them (or a model-independent random subset). The selection bias comes not from the initial model predictions/selection, but instead the (model-independent) selection of the examples to be labeled.

$\endgroup$
1
  • $\begingroup$ Thanks, that's very helpful. The keyword 'pseudo-labeling' is leading me to more literature, I didn't know that term before. $\endgroup$
    – thzu
    May 24, 2022 at 8:46
4
$\begingroup$

The answer is: It depends.

What you describe is a strategy often used to save time and costs for labelling data. It is important that the data you have already labelled (the 20%) is representative of the rest of data (the 80%). At the very least, you must have all classes in those 20%.

It is also important that you select a good detection model to have reliable predictions. Faster RCNN should be a good choice.

However, whether 20% labelled data is enough is difficult to tell. It depends on your data.

Your strategy itself is common. I'd just try whether 20% labelled data is enough. You can also fine-tune your faster RCNN model in between, say after 40% of the data is labelled to improve predictions further.

$\endgroup$
1
  • $\begingroup$ Thanks, continuing training as an intermediate step is a good idea. $\endgroup$
    – thzu
    May 24, 2022 at 8:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .