Let's say I've trained a CNN that is predicting/inferring live samples that it hasn't seen before. In the event the network makes a correct prediction, would including this as a new sample in its training set increase the model accuracy even further when re-training the network?

I'm unsure about this since it seems as though the network has already learnt the necessary features for making the correct prediction, so adding it as a new training sample might be redundant. On the other hand it might also reinforce to the network that it's on the right track, perhaps giving it further confidence to generalize with whatever features its learnt in regards to that class, that it might be able to apply to the same class in other images it might otherwise make an incorrect prediction with?

The reason I'm thinking of this is that manually labeling each image is a time-consuming process, however if a simple "Correct/Incorrect" popup box was presented after the network made a live prediction, then it's simply a matter of clicking a single button to generate a new labelled training sample, which would be a far easier labeling task.

So how useful would it be to do something like this?

  • $\begingroup$ You ask at the top and in the title whether adding correct samples only would help, but your plan includes collecting data for both true and false samples. Is there a reason why you are asking about focusing only on correct samples? Is this a multi-class classifier and you don't want to bother users with assigning correct label just yes/no? $\endgroup$ – Neil Slater Apr 24 '19 at 6:49
  • $\begingroup$ yes thats correct, but Im the only user. I just want to label the model as quickly as possible since its the bottleneck for me rather than getting the data. ie: if I could collect 1000 images easily that my model can predict with 90% accuracy, its much faster for me to click yes/no on each of them and get 900 correctly labelled images, than manually labelling them all. The thing is if I do that, will my accuracy increase past 90%? Ie: after doing all that, will the "no" images have a higher chance of being correct after re-training? $\endgroup$ – user4779 Apr 24 '19 at 8:37

Your suggestion is risky. It might make improvements to your classifier, but it may also reduce generalisation.

The two conflicting factors in play are:

  • Adding data points to a model which has capacity to learn more detail can improve its performance.

  • Training from a different distribution of data points than your target population can reduce its performance.

From the first point, you need your model to be able to accept the new data. This can be harder to achieve than you might think at first - if you have tuned some regularisation parameters by using cross-validation, then you may have at least in part saturated the model's capacity in order to prevent over-fitting. That means fitting to new data could require re-tuning your hyper-parameters as well - but you probably won't need to start from scratch, just search nearby.

It is hard to tell the impact from the second point. There is not going to be a general answer here, too much depends on how the data is arranged for your problem. My gut feeling is that you will notice an improvement to the model initially, but that there will be diminishing returns due to the self-selection of already-correct data points.

You should definitely keep back a cross-validation set of data distributed as it is in production, that has been properly and fully labelled in order to assess this work. This will be your way to assess whether generalisation is improving using your approach.

Worst case: You may need to go back and re-label the mistakes in order to significantly improve performance of your model. So I would suggest part of your auto-labelling process should store the misclassified items somewhere so you can re-visit them and spend the extra effort.

Sadly, even collecting more properly labelled data is not a guaranteed fix - some models already have enough data. You can check whether that might applies in your case by training with different sizes of training set (from the available training data you have so far), and seeing what the trend in performance is when you increase the training set size. In fact, you should probably do this first, before you invest significant effort in collecting more data.

  • $\begingroup$ Good answer and thank you - I completely missed the fact that removing some samples conditionally would of course change the distribution.A question though - would this be a bad thing in the reverse case? Ie: If I specifically sought out to find incorrect predictions and include those in the training set? This would still be a different distribution I'd imagine but would your answer still apply in this scenario? $\endgroup$ – user4779 Apr 25 '19 at 3:47
  • 1
    $\begingroup$ @user4779: Yes my answer would still apply. There would be a difference in behaviour - what the NN got better at, and worse at due to that selection. I have tried something similar in the past - re-training progressively by selecting error cases - and in my case it performed very badly. Of course, as always, actual results depend on your problem. $\endgroup$ – Neil Slater Apr 25 '19 at 6:36
  • $\begingroup$ Thanks again Neil. One final question if you have the time. If it performs badly on some samples, would the correct strategy be to find something about the error samples that might be unique, and then manually capture more training samples that are similar to those error cases, labeling and including all of them regardless of whether the network predicts them correctly or not? $\endgroup$ – user4779 Apr 25 '19 at 7:01
  • 1
    $\begingroup$ @user4779: Again it will depend. With that description, you are heading towards error analysis, which usually requires some domain knowledge. E.g. mlwiki.org/index.php/Error_Analysis $\endgroup$ – Neil Slater Apr 25 '19 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.