Dropout causes too much noise for network to train

Question

I am using dropout of different values to train my network. The problem is, dropout is contributing almost nothing to training, either causing so much noise the error never changes, or seemingly having no effect on the error at all:

The following runs were seeded.

key: dropout = 0.3, means 30% chance of dropout

graph x axis: iteration

y axis: error

dropout=0

dropout = 0.001

dropout = 0.1

dropout = 0.5

I don't quite understand why dropout of 0.5 effectively kills the networks ability to train. This specific network here is rather small, a CNN of architecture:

3x3x3                    Input image
3x3x3                    Convolutional layer: 3x3x3, stride = 1, padding = 1
20x1x1                   Flatten layer: 27 -> 20
20x1x1                   Fully connected layer: 20
10x1x1                   Fully connected layer: 10
2x1x1                    Fully connected layer: 2

But I have tested a CNN with architecture:

10x10x3                  Input image
9x9x12                   Convolutional layer: 4x4x12, stride = 1, padding = 1
8x8x12                   Max pooling layer: 2x2, stride = 1
6x6x24                   Convolutional layer: 3x3x24, stride = 1, padding = 0
5x5x24                   Max pooling layer: 2x2, stride = 1
300x1x1                  Flatten layer: 600 -> 300
300x1x1                  Fully connected layer: 300
100x1x1                  Fully connected layer: 100
2x1x1                    Fully connected layer: 2

overnight with dropout = 0.2 and it completely failed to learn anything, having an accuracy of just below 50%, whereas without dropout, its accuracy is ~85%. I would just like to know if there's a specific reason as to why this might be happening. My implementation of dropout is as follows:

activation = relu(val)*(random.random() > self.dropout)

then at test time:

activation = relu(val)*(1-self.dropout)

Answer 1

why dropout of 0.5 effectively kills the networks ability to train.

because that is too mutch normal values are like 0.15-0.05. Imagine, the 50% of input image is randomly set to 0, THEN it happens on next layer, means in average 25% remains, etc... also if you have small dataset with too different images for each class, this + drouput wil confuse the network. Also your CNN setup is not realy rational. Too much fc layers, replace one or two fc to convo layers. And i d say the reason of using 4x4 is only with stride 2, else use 3x3. And you sould use batch normalisation and augmentation like small noice would be probably petter then dropout in your case.

Answer 2

Your issue may be your loss function.

I have been using dropout though to address a regression neural network that I wish to output coordinates (of a 10x10 grid)

---architecture---
10x10x3                  Input image
...
100x1x1                  Fully connected layer: 100
2x1x1                    Fully connected layer: 2
------------------

It appears like your network is outputting an $(x, y)$ coordinate in the grid, and you are using the mean-squared error as your loss function. Dropout on the final layer will greatly increase the variance of your output, and the loss. In particular, even if your model at test-time is perfect, your training standard deviation is still at least $$\sqrt{\frac{\overbrace{100}^\text{final hidden dimension} \text{dropout}\cdot(1-\text{dropout})}{\text{batch_size}}}=70\%$$ of the correct value for a dropout of 0.2 and batch size of 32. My recommendations are

1. Treat this as a classification problem: softmax $10\times 10$ outputs, asking which pixel is correct.
2. Do not use dropout on the final layer.
3. If you can get more data, use a larger batch size.