Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
Let's assume I have a CNN model trained to categorize some objects on the images. By using this model I find more categorized images. If I now retrain this model on data set that consists old set and newly categorised images is there a chance that such new model will have higher accuracy? Or maybe because new data posses only information that could be found on initial set, model will have similar/lower accuracy?
Please let me know if something unclear.
The most likely outcome of this approach is wasted time and very little effect on accuracy.
There will be changes to the model. Some will be beneficial and improve the model, but some will backfire making it worse.
For instance:
The model predicts with probability 0.4 that an image is in a certain class. It is the highest prediction, and actually true. It will be added to the training dataset with a "ground truth" of probability 1.0, so on balance more and better data has been added to the data set. This will improve generalisation, as whatever caused the relatively low 0.4 value initially - e.g. a pose or lighting variation - will now be covered correctly in the training set.
The model predicts with probability 0.4 that an image is in a certain class. It is the highest prediction, and actually false. It will be added to the dataset with "ground truth" of probability 1.0 for the wrong class. This will weaken associations to the correct class for similar input images, meaning for exaple that a certain pose or lighting difference that is already causing problems for the model will be used to incorrectly classify images in future.
These two scenarios will occur, on average, at a rate determined by the model's current test accuracy. So if your current model is 90% accurate, 1 in 10 images in your new training data will be mislabelled. This will "lock in" the current errors at the same rate on average as they already occur.
The effect may be a drift up or down in accuracy as the model will definitely change due to the new training data. However you have little to no control over how this drift effect goes if you are not willing or able to oversee the automatic classifications generated on new data by the model.
There are a few ways to get some improvement unsupervised from new data. For instance:
Build an autoencoder from the early convolutional layers of your model and train it to re-generate all inputs as outputs. This should help it learn important features of the variations in data that you are using. Once this training is done, discard the decoder part of the auto-encoder and add your classifier back in to fine tune it. This may help if you have only a small amount of labelled data, but a lot of unlabelled data.
Use a model that has better accuracy than yours to auto-label the data. This might seem a little chicken-and-egg, but you may be able to create such a model using ensemble techniques. The ensemble model could be too awkward to use in production, but may still be used in an auto-labeling pipeline to improve your training data.
Note you may get even better results simply ignoring the extra unlabeled data, and instead fine tuning a high quality ImageNet-trained model on the labeled data you already have - saving yourself a lot of effort. Depends on the nature of the images, and how much labeled data you are already working with.
