I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly having no effect on the error at all:

The following runs were seeded.

key: dropout = 0.3, means 30% chance of dropout

graph x axis: iteration

y axis: error

dropout=0 enter image description here

dropout = 0.001enter image description here

dropout = 0.1enter image description here

dropout = 0.5 enter image description here

I don't quite understand why dropout of 0.5 effectively kills the networks ability to train. This specific network here is rather small, a CNN of architecture:

3x3x3                    Input image
3x3x3                    Convolutional layer: 3x3x3, stride = 1, padding = 1
20x1x1                   Flatten layer: 27 -> 20
20x1x1                   Fully connected layer: 20
10x1x1                   Fully connected layer: 10
2x1x1                    Fully connected layer: 2

But I have tested a CNN with architecture:

10x10x3                  Input image
9x9x12                   Convolutional layer: 4x4x12, stride = 1, padding = 1
8x8x12                   Max pooling layer: 2x2, stride = 1
6x6x24                   Convolutional layer: 3x3x24, stride = 1, padding = 0
5x5x24                   Max pooling layer: 2x2, stride = 1
300x1x1                  Flatten layer: 600 -> 300
300x1x1                  Fully connected layer: 300
100x1x1                  Fully connected layer: 100
2x1x1                    Fully connected layer: 2

overnight with dropout = 0.2 and it completely failed to learn anything, having an accuracy of just below 50%, whereas without dropout, its accuracy is ~85%. I would just like to know if there's a specific reason as to why this might be happening. My implementation of dropout is as follows:

activation = relu(val)*(random.random() > self.dropout)

then at test time:

activation = relu(val)*(1-self.dropout)

  • $\begingroup$ Is your dataset small? $\endgroup$
    – user9947
    Jul 16, 2019 at 4:05
  • $\begingroup$ @DuttaA I'm using batch sizes of 32 on a data set of about 1000 images $\endgroup$
    – Recessive
    Jul 16, 2019 at 4:12
  • $\begingroup$ Dropout is an effective way to utilize an overparameterized model by averaging multiple models that cover subsets of an overarching feature space. In practice its used as a tool against overfitting. In your case you dont even mention test set error. I would reccomend looking more into what dropout is, and what you want from using it. $\endgroup$
    – mshlis
    Jul 16, 2019 at 4:14
  • $\begingroup$ See the above comment, to make it more clear I very large CNNs, it very hard to stop the CNN from over fitting on the training set and so we use dropout, but in smaller CNN it might actually degrade performance (although I am not sure if such drastic effects would occur) $\endgroup$
    – user9947
    Jul 16, 2019 at 4:29
  • $\begingroup$ @mshlis Sorry, I didn't specify, when I said " having an accuracy of just below 50%, whereas without dropout, its accuracy is ~85% " I was referring to test set error. I took an accurate classification as above 0.65 for the correct output node (with a softmax activation). I have been using dropout though to address a regression neural network that I wish to output coordinates (of a 10x10 grid), but instead seems to just take the average (x, y) coordinates of the training data, which I was guessing was over fitting, hence I am trying dropout. $\endgroup$
    – Recessive
    Jul 16, 2019 at 4:41

1 Answer 1

why dropout of 0.5 effectively kills the networks ability to train.

because that is too mutch normal values are like 0.15-0.05. Imagine, the 50% of input image is randomly set to 0, THEN it happens on next layer, means in average 25% remains, etc... also if you have small dataset with too different images for each class, this + drouput wil confuse the network. Also your CNN setup is not realy rational. Too much fc layers, replace one or two fc to convo layers. And i d say the reason of using 4x4 is only with stride 2, else use 3x3. And you sould use batch normalisation and augmentation like small noice would be probably petter then dropout in your case.

  • $\begingroup$ From the paper that presented this technique: "In the simplest case, each unit is retained with a fixed probability p independent of other units, where p can be chosen using a validation set or can simply be set at 0.5, which seems to be close to optimal for a wide range of networks and tasks", jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf introduction, paragraph 4 $\endgroup$
    – Recessive
    Jul 22, 2019 at 3:05
  • $\begingroup$ Why are you creating a custom dropout. Why not use the keras dropout layer after each dense layer except of course not the last layer? Please provide actual code you used to create the model. $\endgroup$
    – Gerry P
    Mar 6, 2020 at 6:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .