What to do if CNN cannot overfit a training set on adding dropout?

Question

I have been trying to use CNN for a regression problem. I followed the standard recommendation of disabling dropout and overfitting a small training set prior to trying for generalization. With a 10 layer deep architecture, I could overfit a training set of about 3000 examples. However, on adding 50% dropout after the fully-connected layer just before the output layer, I find that my model can no longer overfit the training set. Validation loss also stopped decreasing after a few epochs. This is a substantially small training set, so overfitting should not have been a problem, even with dropout. So, does this indicate that my network is not complex enough to generalize in the presence of dropout? Adding additional convolutional layers didn't help either. What are the things to try in this situation? I will be thankful if someone can give me a clue or suggestion.

PS: For reference, I am using the learned weights of the first 16 layers of Alexnet and have added 3 convolutional layers with ReLU non-linearity followed by a max pooling layer and 2 fully connected layers. I update weights of all layers during training using SGD with momentum.

Answer 1

Sorry if this is a bad use of answer to add comment but since my reputation is not high enough this is only way to leave a comment to OP's question.

I think some of the answers misunderstood the OP's intention.

Over fitting is used as a means to test the complexity of the model - if a model cannot overfit a small dataset then it's likely not able to generalize well.

It's not that OP misunderstood the meaning of over fitting.

For Instance, I think this discussion is relevant: https://stats.stackexchange.com/questions/492165/what-to-do-when-a-neural-network-cannot-overfit-one-training-sample

Answer 2

Let's start with understanding what over-fitting means. Your model is over-fitting if during training your training loss continues to decrease but (in the later epochs) your validation loss begins to increase. That means the model can not generalize well to images it has not previously encountered.

Naturally, you do not want this situation. What you want is a high training accuracy and a very low validation loss, which implies a high validation accuracy.

The first task is to ensure that your model gets a high training accuracy. Once that is accomplished, you can work on getting a low validation loss.

If your model is overfitting, there are several ways to mitigate the problem. First, start out with a simple model. If you have a lot of dense layers with a lot of neurons, reduce the hidden dense layers to a minimum. Typical just leave the top dense layer used for final classification. Then see how the model trains. If it trains well, look at the validation loss and see if it is reducing in the later epochs. If the model does not train well, add a dense layer followed by a dropout layer. Use the level of dropout to adjust for overfitting. If it still trains poorly, increase the number of neurons, and train again. If that fails, add another dense hidden layer with fewer neurons than the previous layer followed by another dropout layer.

Another method to combat overfitting is to add regularizers to the dense layers. Documentation for that is here.

Answer 3

I think that your misuse of the term over-fitting made the question vague. In layman terms, over-fitting means that a model fails to generalize to real-world scenarios, but is accurate with the training set.

Using a dropout layer means that the network cuts down on neurons that are used for training, in this case, 50%.

Recommendations for improving training accuracy would be:

• Transfer learning
• Adding more layers to the network (also shifting number of neurons helps)
• Adding epochs
• Changing optimizer (Adam and RMSProp are some of my suggestions)
• Adding activation layers