Is this overfitting avoidable?

Question

I am trying a modification of Mobilenet in which I add feedback from the softmax layer into the early layers (to implement this I put a second net after the first, which receives connections from the softmax layer of the first, the pretrained weights being non trainable). The idea was to mimic the massive feedback projections in the brain, which presumably could help object recognition by enhancing specific filters and inhibiting others.

I took the pretrained network from Keras and started to retrain it on Imagenet. I noticed that the training accuraccy increased right in the first epoch. My computer is very slow thus I cannot train for too long, an epoch takes 3.5 days. So after an epoch I tried the validation set, but instead the accuracy went down to almost half that of the pretrained values.

My question is if this is and obvious case of overfitting. That is, will continued training increase the accuracy of the training set at the expense of the validation set, or is this a normal behavior expected at the initial stages of training, so that if I keep training for a few more epochs I could expect the validation set accuracy to go eventually up? Any ideas that could help are welcomed.

Answer 1

an epoch takes 3.5 days

First of all, use colab to iterate quickly, it offers unlimited 12 hours of free GPU.

to retrain it on Imagenet.

That said, we could use complex models without being afraid of overfitting due to large training size.

will continued training increases the accuracy of the training set at the expense of the validation set

In many cases that's the case.

The idea was to mimic the massive feedback projections in the brain.

My suggestion is to read the third part of the deeplearning book which includes Representation Learning, Structured Probabilistic Models for Deep Learning, Monte Carlo Methods, Confronting the Partition Function, Approximate Inference, Deep Generative Models.

Part III is the most important for a researcher—someone who wants to understand the breadth of perspectives that have been brought to the field of deep learning, and push the field forward towards true artificial intelligence.