I have been trying to use CNN for a regression problem. I followed the standard recommendation of disabling dropout and overfitting a small training set prior to trying for generalization. With a 10 layer deep architecture, I could overfit a training set of about 3000 examples. However, on adding 50% dropout after the fully-connected layer just before the output layer, I find that my model can no longer overfit the training set. Validation loss also stopped decreasing after a few epochs. This is a substantially small training set, so overfitting should not have been a problem, even with dropout. So, does this indicate that my network is not complex enough to generalize in the presence of dropout? Adding additional convolutional layers didn't help either. What are the things to try in this situation? I will be thankful if someone can give me a clue or suggestion.

PS: For reference, I am using the learned weights of the first 16 layers of Alexnet and have added 3 convolutional layers with ReLU non-linearity followed by a max pooling layer and 2 fully connected layers. I update weights of all layers during training using SGD with momentum.

  • 2
    $\begingroup$ Can you overfit with no dropout but half the neurons? Can you overfit with double the amount of neurons and 50% dropout? $\endgroup$ – Martin Thoma Oct 14 '18 at 16:49
  • 1
    $\begingroup$ What do you think “overfit” means? I suspect that you’re using the term incorrectly. In particular, overfitting is bad. $\endgroup$ – Stella Biderman Oct 15 '18 at 2:45
  • $\begingroup$ What's the size of your output layer? Also, I guess with overfitting you imply zero training error?? I would suggest adding dropout before the last hidden layer. P.S. writing as an answer, because I am unable to post comments $\endgroup$ – Karan Dhingra Sep 4 at 8:28

Let's start with understanding what over-fitting means. Your model is over-fitting if during training your training loss continues to decrease but (in the later epochs) your validation loss begins to increase. That means the model can not generalize well to images it has not previously encountered.

Naturally, you do not want this situation. What you want is a high training accuracy and a very low validation loss, which implies a high validation accuracy.

The first task is to ensure that your model gets a high training accuracy. Once that is accomplished, you can work on getting a low validation loss.

If your model is overfitting, there are several ways to mitigate the problem. First, start out with a simple model. If you have a lot of dense layers with a lot of neurons, reduce the hidden dense layers to a minimum. Typical just leave the top dense layer used for final classification. Then see how the model trains. If it trains well, look at the validation loss and see if it is reducing in the later epochs. If the model does not train well, add a dense layer followed by a dropout layer. Use the level of dropout to adjust for overfitting. If it still trains poorly, increase the number of neurons, and train again. If that fails, add another dense hidden layer with fewer neurons than the previous layer followed by another dropout layer.

Another method to combat overfitting is to add regularizers to the dense layers. Documentation for that is here.

| improve this answer | |

I think that your misuse of the term over-fitting made the question vague. In layman terms, over-fitting means that a model fails to generalize to real-world scenarios, but is accurate with the training set.

Using a dropout layer means that the network cuts down on neurons that are used for training, in this case, 50%.

Recommendations for improving training accuracy would be:

  • Transfer learning
  • Adding more layers to the network (also shifting number of neurons helps)
  • Adding epochs
  • Changing optimizer (Adam and RMSProp are some of my suggestions)
  • Adding activation layers
| improve this answer | |
  • $\begingroup$ How did you determine your network was over fitting? Did you use a validation set and if you did did the validation error start to increase as you increased the number of epochs? You need to be sure you are over fitting before you start to change things. Can you provide the training loss and validation loss data for each epoch. That will help to tell what is going on. $\endgroup$ – Gerry P Mar 6 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.