Is it possible that the fine-tuned pre-trained model performs worse than the original pre-trained model?

Question

I have downloaded a pre-trained EfficientDet D2 model (Tensorflow 2.0) and trained it on some data (about 20000 images with 20 classes). I set the number of steps to 25000 and batch size to 3 (computer resources are not the best).

However, if I try to make predictions, the pre-trained model makes better predictions than the model I have trained on the additional data. Is this expected behaviour?

For example, an image of a person may be 78% accurate on the pre-trained model and only 54% accurate on the same image when trained.

Answer 1

Yes, this is quite the expected behavior. The main difference between expected and current behavior lies in the amount of data you are using for training VS the amount of data that the pre-trained model was trained with.

Take into account that pre-trained models have been trained over popular datasets, the most common ones are: COCO, ImageNet and Open Images. And the amount of data differs:

• COCO: 330K images
• ImageNet: 1.5M images
• OpenImages: 9M images
• Your dataset: 20K images

You could say, well, but I started from a pre-trained model, so the network should already know how to detect a person. Well that is true, but you are training it again, you are not using transfer learning (freezing backbone layers, or adding extra heads / extra channels to detect other features or other classes). So what is happening is that your model, even when it started from good pre-trained weights, is fitting to your 20K images data.

As I see it, you have two options: either increase your dataset size or use transfer leanring.