I have been trying to use CNN for a regression problem. I followed the standard recommendation of disabling dropout and overfitting a small training set prior to trying for generalization. With a 10 layer deep architecture, I could overfit a training set of about 3000 examples. However, on adding 50% dropout after the fully-connected layer just before the output layer, I find that my model can no longer overfit the training set. Validation loss also stopped decreasing after a few epochs. This is a substantially small training set, so overfitting should not have been a problem, even with dropout. So, does this indicate that my network is not complex enough to generalize in the presence of dropout? Adding additional convolutional layers didn't help either. What are the things to try in this situation? I will be thankful if someone can give me a clue or suggestion.

PS: For reference, I am using the learned weights of the first 16 layers of Alexnet and have added 3 convolutional layers with ReLU non-linearity followed by a max pooling layer and 2 fully connected layers. I update weights of all layers during training using SGD with momentum.

  • 1
    $\begingroup$ Can you overfit with no dropout but half the neurons? Can you overfit with double the amount of neurons and 50% dropout? $\endgroup$ – Martin Thoma Oct 14 '18 at 16:49
  • $\begingroup$ What do you think “overfit” means? I suspect that you’re using the term incorrectly. In particular, overfitting is bad. $\endgroup$ – Stella Biderman Oct 15 '18 at 2:45

At first we should define, what overfitting means. The OP has used 3000 training examples for reducing the error-rate of his neural network. Overfitting is happening, if it is not possible to use the learned model on new data. That means, the error rate for data outside of training-data is high. Overcome the problem is not possible. Because today's hardware like the Nvidia GPU have a limited performance, so it is not possible to extend the number of training examples to 3 million or more. And a magic option, which can be selected in the convolutional neural network is also not available which increases the ability to generalize. So it is a very good example for a failed software project. At the beginning, the user was confident, that with a neural network the problems can be solved, but in the concrete example the algorithm didn't work.

Instead of leaving the user alone with a pessimistic outlook, I want to give some hints what is possible to fix the problem. Usually, a bad working neural network can be overcome with manual implemented heuristics. That is knowledge which depends on the subject. For example, if the aim is to detect cats in images, than a cat-ontology is helpful, if the aim is to predict a weather-timeseries, than a RDF-database with weather information can be used. In both cases, the problem is escalated to the domain of knowledge-processing, which is combined with neural networks. Heuristics helps to reducing the state-space.


Think the use of over fitting made the question vague. Over fitting in layman terms means that a model fails to generalize real world scenarios but is accurate with the training set.
Using a dropout layer means that the network cuts down on neurons that are used for training in this case 50%.
Recommendations for improving training accuracy would be:

  1. Transfer learning
  2. Adding more layers to the network (also shifting number of neurons helps)
  3. Adding epochs
  4. Changing optimizer (Adam and RMSProp are some of my suggestions)
  5. Adding activation layers

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.