Is it possible that the fine-tuned pre-trained model performs worse than the original pre-trained model?

I have downloaded a pre-trained EfficientDet D2 model (Tensorflow 2.0) and trained it on some data (about 20000 images with 20 classes). I set the number of steps to 25000 and batch size to 3 (computer resources are not the best).

However, if I try to make predictions, the pre-trained model makes better predictions than the model I have trained on the additional data. Is this expected behaviour?

For example, an image of a person may be 78% accurate on the pre-trained model and only 54% accurate on the same image when trained.

• Hello. Welcome to AI SE! I changed the title of your post to put there what I think is your question. Please, make sure the current version of your post is still consistent with your original post.
– nbro
Jul 14, 2021 at 11:31

Yes, this is quite the expected behavior. The main difference between expected and current behavior lies in the amount of data you are using for training VS the amount of data that the pre-trained model was trained with.

Take into account that pre-trained models have been trained over popular datasets, the most common ones are: COCO, ImageNet and Open Images. And the amount of data differs:

• COCO: 330K images
• ImageNet: 1.5M images
• OpenImages: 9M images
• I am using pytorch for two years now, but last time I checked you needed to iterate the model layers and set the target layers to trainable=false