6
$\begingroup$

Why is dropout favored compared to reducing the number of units in hidden layers for the convolutional networks?

If a large set of units leads to overfitting and dropping out "averages" the response units, why not just suppress units?

I have read different questions and answers on the dropout topic including these interesting ones, What is the "dropout" technique? and this other Should I remove the units of a neural network or increase dropout?, but did not get the proper answer to my question.

By the way, it is weird that this publication A Simple Way to Prevent Neural Networks from Overfitting (2014), Nitish Srivastava et al., is cited as being the first on the subject. I have just read one that is from 2012: Improving neural networks by preventing co-adaptation of feature detectors.

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
6
$\begingroup$

Dropout only ignores a portion of units during a single training batch update. Each training batch will use a different combination of units which gives them the best chance of that portion of them working together to generalize. Note the weights for each unit are kept and will be updated during the next batch in which that unit is selected. During inference, yes, all units are used (with a factor applied to activation...the same factor that defines the fraction used)...this becomes essentially an ensemble of all the different combinations of units that were used.

Contrasted with fewer units, the fewer units approach will only learn what those fewer units can be optimized for. Think of dropout as an ensemble of layers of fewer units.(with the exception that there are partial weight sharing between the layers)

Improve this answer
$\endgroup$
1
  • $\begingroup$ I believe the comment about "each training batch" using a "different combination of units" is incorrect, at least as per the original paper on dropout. The dropout is applied per observation, not per mini batch. I believe this also means that weights of units that are dropped for only some of the observations in a mini batch are indeed updated during back propagation because they contribute to the loss function, which is an average across all observations in a mini batch. See: stats.stackexchange.com/questions/335690/…. $\endgroup$
    – Fijoy Vadakkumpadan
    Jan 2 at 3:36
2
$\begingroup$

The idea of dropout is that, at training time, with a certain probability $p_i \in [0, 1]$, the unit (or neuron) $i$ is dropped, $\forall i$, that is, the output of unit $i$ is set to zero so that $i$ does not affect the other units it is connected to, both during the forward and backward (or back-propagation) passes (or steps). At every mini-batch, you randomly drop usually different units, so, across different mini-batches (and consequently epochs), you do not always or necessarily drop the same units.

The title of the paper Improving neural networks by preventing co-adaptation of feature detectors emphasizes that dropout prevents the co-adaptation of the units (the feature detectors), so units attempt to detect certain features independently of other units, which reduces overfitting, that is, it improves the generalization ability of the neural network.

At test time, no unit is usually dropped. However, there is an approximation of a deep Gaussian process and Bayesian neural network that is based on the application of dropout at training and test times. This is called Monte Carlo dropout or, in short, MC dropout, for reasons you can understand if you read the paper Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.

There's also the possibility to drop the connections between the neurons, which is called DropConnect, rather than the neurons themselves. These two approaches are slightly different, even though DropConnect can be seen as a generalization of dropout. In DropConnect, you do not switch off completely the units, but only the contributions of certain units to the output of other units. In dropout, you completely switch off certain units.

If you decided to deterministically (and manually) reduce the number of units before training, essentially, you would train a fixed smaller network, but this will not necessarily reduce overfitting or, more precisely, co-adaptation of the units. In dropout, you randomly select the units to drop, so, at every mini-batch (or epoch, depending on the implementation of dropout), you effectively train a random subset of the units of the original neural network and, because of this, it can be thought of as an ensemble of smaller neural networks.

The two papers Improving neural networks by preventing co-adaptation of feature detectors (2012) and Dropout: A Simple Way to Prevent Neural Networks from Overfitting (2014) have exactly the same authors, but the latter paper was published in the Journal of Machine Learning Research, while the former wasn't apparently published in any journal. In fact, Dropout: A Simple Way to Prevent Neural Networks from Overfitting does not even cite Improving neural networks by preventing co-adaptation of feature detectors, but it cites the master's thesis Improving Neural Networks with Dropout (2013) by Nitish Srivastava, who is one of the authors of dropout.

Improve this answer
$\endgroup$
2
  • $\begingroup$ (+1) But not that it's not necessarily true that different units are dropped at each training iteration. Vanilla dropout is essentially a Bernoulli(p) binary mask, so the same unit can be dropped during 2 successive iterations with probability $p^2$. Likewise, the entire dropout mask could, by chance, be the same between two iterations with some small probability. $\endgroup$
    – Sycorax
    Dec 12, 2019 at 22:10
  • $\begingroup$ @SycoraxsaysReinstateMonica Thanks for the feedback. I've started this answer with "The idea of dropout", because I wanted more to give the intuition. However, you're right, the same units could be dropped during two successive training iterations. I've edited my answer to try accommodating the information in your comment. $\endgroup$
    – nbro
    Dec 12, 2019 at 22:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .