1
$\begingroup$

I am trying a modification of Mobilenet in which I add feedback from the softmax layer into the early layers (to implement this I put a second net after the first, which receives connections from the softmax layer of the first, the pretrained weights being non trainable). The idea was to mimic the massive feedback projections in the brain, which presumably could help object recognition by enhancing specific filters and inhibiting others.

I took the pretrained network from Keras and started to retrain it on Imagenet. I noticed that the training accuraccy increased right in the first epoch. My computer is very slow thus I cannot train for too long, an epoch takes 3.5 days. So after an epoch I tried the validation set, but instead the accuracy went down to almost half that of the pretrained values.

My question is if this is and obvious case of overfitting. That is, will continued training increase the accuracy of the training set at the expense of the validation set, or is this a normal behavior expected at the initial stages of training, so that if I keep training for a few more epochs I could expect the validation set accuracy to go eventually up? Any ideas that could help are welcomed.

$\endgroup$
1
+50
$\begingroup$

an epoch takes 3.5 days

First of all, use colab to iterate quickly, it offers unlimited 12 hours of free GPU.

to retrain it on Imagenet.

That said, we could use complex models without being afraid of overfitting due to large training size.

will continued training increases the accuracy of the training set at the expense of the validation set

In many cases that's the case.

The idea was to mimic the massive feedback projections in the brain.

My suggestion is to read the third part of the deeplearning book which includes Representation Learning, Structured Probabilistic Models for Deep Learning, Monte Carlo Methods, Confronting the Partition Function, Approximate Inference, Deep Generative Models.

Part III is the most important for a researcher—someone who wants to understand the breadth of perspectives that have been brought to the field of deep learning, and push the field forward towards true artificial intelligence.

$\endgroup$
  • $\begingroup$ thanks for the book suggestion, I will read it. I was not sure to understand your answer. Large data should not result on over fitting, but that is what I get. So could continued training improve the validation accuracy or not (it is lower after training for an epoch?). Also I have a 1080, it should be about the same speed as the k80 used in colab, right? $\endgroup$ – Hugh Mungus May 18 '18 at 15:37
  • $\begingroup$ The 1050ti should be the same speed as the k80, Large data should not result on overfitting BUT what if you have a very very very large model? "with a second net after the first" even with non trainable parameters. by the way, your approach is somehow ill-defined. if you insist to work on some crazy idea, consider toy problems with small (synthetic) dataset and iterate from there. All researcher try their experiments with a suitable computing/time budget. $\endgroup$ – Fadi Bakoura May 21 '18 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.