# Dropout causes too much noise for network to train

I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly having no effect on the error at all:

The following runs were seeded.

key: dropout = 0.3, means 30% chance of dropout

graph x axis: iteration

y axis: error

dropout=0

dropout = 0.001

dropout = 0.1

dropout = 0.5

I don't quite understand why dropout of 0.5 effectively kills the networks ability to train. This specific network here is rather small, a CNN of architecture:

3x3x3                    Input image
3x3x3                    Convolutional layer: 3x3x3, stride = 1, padding = 1
20x1x1                   Flatten layer: 27 -> 20
20x1x1                   Fully connected layer: 20
10x1x1                   Fully connected layer: 10
2x1x1                    Fully connected layer: 2


But I have tested a CNN with architecture:

10x10x3                  Input image
9x9x12                   Convolutional layer: 4x4x12, stride = 1, padding = 1
8x8x12                   Max pooling layer: 2x2, stride = 1
6x6x24                   Convolutional layer: 3x3x24, stride = 1, padding = 0
5x5x24                   Max pooling layer: 2x2, stride = 1
300x1x1                  Flatten layer: 600 -> 300
300x1x1                  Fully connected layer: 300
100x1x1                  Fully connected layer: 100
2x1x1                    Fully connected layer: 2


overnight with dropout = 0.2 and it completely failed to learn anything, having an accuracy of just below 50%, whereas without dropout, its accuracy is ~85%. I would just like to know if there's a specific reason as to why this might be happening. My implementation of dropout is as follows:

activation = relu(val)*(random.random() > self.dropout)

then at test time:

activation = relu(val)*(1-self.dropout)

– user9947
Jul 16 '19 at 4:05
• @DuttaA I'm using batch sizes of 32 on a data set of about 1000 images Jul 16 '19 at 4:12
• Dropout is an effective way to utilize an overparameterized model by averaging multiple models that cover subsets of an overarching feature space. In practice its used as a tool against overfitting. In your case you dont even mention test set error. I would reccomend looking more into what dropout is, and what you want from using it. Jul 16 '19 at 4:14
• See the above comment, to make it more clear I very large CNNs, it very hard to stop the CNN from over fitting on the training set and so we use dropout, but in smaller CNN it might actually degrade performance (although I am not sure if such drastic effects would occur)
– user9947
Jul 16 '19 at 4:29
• @mshlis Sorry, I didn't specify, when I said " having an accuracy of just below 50%, whereas without dropout, its accuracy is ~85% " I was referring to test set error. I took an accurate classification as above 0.65 for the correct output node (with a softmax activation). I have been using dropout though to address a regression neural network that I wish to output coordinates (of a 10x10 grid), but instead seems to just take the average (x, y) coordinates of the training data, which I was guessing was over fitting, hence I am trying dropout. Jul 16 '19 at 4:41

why dropout of 0.5 effectively kills the networks ability to train.