I have 6600 images and I am supposed to know the rotation of the object in each image. So, given an image, I want to regress to a single value.
My attempt: I use Resnet-18 to extract a feature vector of length 1000 from an image. This is then passed to three fully-connected layers: fc(1000, 512) -> fc(512, 64) -> fc(64, 1)
The problem I am facing right now is that my training loss and validation loss immediately go down after the first 5 epochs and then they barely change. But my training and validation accuracy fluctuates wildly throughout.
I understand that I am experiencing over-fitting and I have done the following to deal with it:
- data augmentation (Gaussian noise and color jittering)
- L1 regularization
So far, nothing seems to be changing the results much. The next thing I haven't tried is reducing the size of my neural net. Will that help? If so, how should I reduce the size?