I know dropout layers are used in neural networks during training to provide a form of regularisation in an attempt to mitigate over-fitting.
Would you not get an increased fitness if you disabled the dropout layers during evaluation of a network?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Keras turns off dropout when computing validation accuracy during training. Also when making predictions $\endgroup$– Gerry PMar 3, 2020 at 17:23
-
$\begingroup$ Follow-up question: ai.stackexchange.com/questions/18397/… $\endgroup$– TobiMar 3, 2020 at 17:49
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Dropout is a technique that helps to avoid overfitting during training. That is, dropout is usually used for training.
units may change in a way that they fix up the mistakes of the other units. This may lead to complex co-adaptations. This, in turn, leads to overfitting because these co-adaptations do not generalize to unseen data.
If you want to evaluate your model, you should turn off all dropout layers. For example, PyTorch's model.eval() does this work.
Note that in some cases dropout can be used for inference, e.g. to add some stochasticity to the output.
More about dropout:
-
$\begingroup$ I did not know PyTorch automatically deactivated drop out layers $\endgroup$– TobiMar 3, 2020 at 16:18
$\begingroup$
$\endgroup$
Dropout is usually disabled at test (or evaluation) time. For example, in Keras, dropout is disabled at evaluation time by default, although you can enable it, if you need to (see below). The purpose of dropout is to decorrelate the units (or feature detectors) so that they learn more robust representations of the data (i.e. a form of regularisation).
However, there's also Monte Carlo (MC) dropout, i.e., you train the network with dropout and you also use dropout at test time in order to get stochastic outputs (i.e. you will get different outputs, for different forward passes, given the same inputs). MC dropout is an approximation of Bayesian inference in deep Gaussian processes, which means that MC dropout is roughly equivalent to a Bayesian neural network.
Does the performance of a model increase if dropout is disabled at evaluation time?
Yes, possibly. However, MC dropout provides an uncertainty measure, which can be useful in certain scenarios (e.g. medical scenarios), where a point estimate (i.e. a single prediction or classification) is definitely not appropriate, but you also need a measure of the uncertainty or confidence of the predictions.