2
$\begingroup$

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is $p$, then we are effectively training with a neural network of size $(1−p)N$, where $N$ is the total number of units in the neural network.

Using this logic, there is no limit how big I can make a network, as long as I proportionately increase dropout, I can always effectively train with the same sized network, and thereby just increasing the number of "independent" models working together, making a larger ensemble model. Thereby improving generalization of the model.

For example, if a network with 2 units already achieves good results in the training set (but not in unseen data -i.e validation or test sets-), also a network with 4 units + dropout 0.5 (ensemble of 2 models), and also a network with 8 units + dropout 0.75 (ensemble of 4 models)... and also a network with 1000 units with a dropout of 0.998 (ensemble of 500 models)!

In practice, it is recommended to keep dropout at $0.5$, which advises against the approach mentioned above. So there seem to be reasons for this.

What speaks against blowing up a model together with an adjusted dropout parameter?

$\endgroup$
1
$\begingroup$

There is no incentive to increase the size of the model for not reason. If a model of size x gives the best possible performance, there is no reason to use a model of size 2*x with 0.5 dropout during training. Usually we want to find the smallest possible model with the best performance. Inflating the model just results in higher computational requirements.

You are basically suggesting to use dropout to allow the network to learn the same features more than once (creating a redundancies in the model). That is not the purpose of dropout. Dropout is used to enable the network break unnecessary correlations which occur in the training set. For example if class 1 is the only one that contains in it features A and B, but all the training samples always feature them together. The dropout process can make the model realize that even just one of them is enough to point to class 1.

$\endgroup$
1
  • $\begingroup$ Right, but I stated only good performance on the training set. The original purpose of Dropout as a regularization method is to improve generalization performance (val/ test sets). In this case, the model expansion has a very good reason to be done (or not? That is my question. But the argument 'the model already performs good' was not a motivation of this question, rather, "the model is not performing well, and overfitting"). $\endgroup$ – hirschme Dec 13 '18 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.