2
$\begingroup$

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is $p$, then we are effectively training with a neural network of size $(1−p)N$, where $N$ is the total number of units in the neural network.

Using this logic, there is no limit how big I can make a network, as long as I proportionately increase dropout, I can always effectively train with the same sized network, and thereby just increasing the number of "independent" models working together, making a larger ensemble model. Thereby improving generalization of the model.

For example, if a network with 2 units already achieves good results in the training set (but not in unseen data -i.e validation or test sets-), also a network with 4 units + dropout 0.5 (ensemble of 2 models), and also a network with 8 units + dropout 0.75 (ensemble of 4 models)... and also a network with 1000 units with a dropout of 0.998 (ensemble of 500 models)!

In practice, it is recommended to keep dropout at $0.5$, which advises against the approach mentioned above. So there seem to be reasons for this.

What speaks against blowing up a model together with an adjusted dropout parameter?

$\endgroup$

2 Answers 2

2
$\begingroup$

There is no incentive to increase the size of the model for not reason. If a model of size x gives the best possible performance, there is no reason to use a model of size 2*x with 0.5 dropout during training. Usually we want to find the smallest possible model with the best performance. Inflating the model just results in higher computational requirements.

You are basically suggesting to use dropout to allow the network to learn the same features more than once (creating a redundancies in the model). That is not the purpose of dropout. Dropout is used to enable the network break unnecessary correlations which occur in the training set. For example if class 1 is the only one that contains in it features A and B, but all the training samples always feature them together. The dropout process can make the model realize that even just one of them is enough to point to class 1.

$\endgroup$
1
  • $\begingroup$ Right, but I stated only good performance on the training set. The original purpose of Dropout as a regularization method is to improve generalization performance (val/ test sets). In this case, the model expansion has a very good reason to be done (or not? That is my question. But the argument 'the model already performs good' was not a motivation of this question, rather, "the model is not performing well, and overfitting"). $\endgroup$
    – hirschme
    Commented Dec 13, 2018 at 18:40
0
$\begingroup$

Increasing layer size while increasing dropout is possible, but will not lead to more effective learning when taken to the extreme. Dropout is a regularization technique which actually makes the model "worse" for the sake of increasing generalization capabilities. With increased dropout you increase the negative impacts of dropout (learning less), while not significantly increasing the upsides (reducing overfitting).

When using double the neurons and increasing dropout respectively, you will have to show the model double the amount of data to learn the same amount, since only half the neurons get updated each iteration. If you always use the same amount of data, this means you would have to show the model each sample twice to achieve the same results. This is much worse than simply training one iteration with less dropout.

Additionally, with high dropout the rate of neurons "firing together" is reduced to a minimum, meaning the neurons are almost independent of each other, resulting in a great loss of computational power.

The highest Dropout values reccommended are no higher than 0.8:

A good value for dropout in a hidden layer is between 0.5 and 0.8. Input layers use a larger dropout rate, such as of 0.8.

https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/

Also check out this question for a more detailed answer.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .