# Should I remove the units of a neural network or increase dropout?

When adding dropout to a neural network, we are randomly removing a fraction of the connections (setting those weights to zero for that specific weight update iteration). If the dropout probability is $$p$$, then we are effectively training with a neural network of size $$(1−p)N$$, where $$N$$ is the total number of units in the neural network.

Using this logic, there is no limit how big I can make a network, as long as I proportionately increase dropout, I can always effectively train with the same sized network, and thereby just increasing the number of "independent" models working together, making a larger ensemble model. Thereby improving generalization of the model.

For example, if a network with 2 units already achieves good results in the training set (but not in unseen data -i.e validation or test sets-), also a network with 4 units + dropout 0.5 (ensemble of 2 models), and also a network with 8 units + dropout 0.75 (ensemble of 4 models)... and also a network with 1000 units with a dropout of 0.998 (ensemble of 500 models)!

In practice, it is recommended to keep dropout at $$0.5$$, which advises against the approach mentioned above. So there seem to be reasons for this.

What speaks against blowing up a model together with an adjusted dropout parameter?