1
$\begingroup$

I know dropout layers are used in neural networks during training to provide a form of regularisation in an attempt to mitigate over-fitting.

Would you not get an increased fitness if you disabled the dropout layers during evaluation of a network?

$\endgroup$
2
$\begingroup$

Dropout is usually disabled at test (or evaluation) time. For example, in Keras, dropout is disabled at evaluation time by default, although you can enable it, if you need to (see below). The purpose of dropout is to decorrelate the units (or feature detectors) so that they learn more robust representations of the data (i.e. a form of regularisation).

However, there's also Monte Carlo (MC) dropout, i.e., you train the network with dropout and you also use dropout at test time in order to get stochastic outputs (i.e. you will get different outputs, for different forward passes, given the same inputs). MC dropout is an approximation of Bayesian inference in deep Gaussian processes, which means that MC dropout is roughly equivalent to a Bayesian neural network.

Does the performance of a model increase if dropout is disabled at evaluation time?

Yes, possibly. However, MC dropout provides an uncertainty measure, which can be useful in certain scenarios (e.g. medical scenarios), where a point estimate (i.e. a single prediction or classification) is definitely not appropriate, but you also need a measure of the uncertainty or confidence of the predictions.

| improve this answer | |
$\endgroup$
2
$\begingroup$

Dropout is a technique that helps to avoid overfitting during training. That is, one can use dropout only for training.

units may change in a way that they fix up the mistakes of the other units. This may lead to complex co-adaptations. This, in turn, leads to overfitting because these co-adaptations do not generalize to unseen data.

If you want to evaluate your model, you should turn off all dropout layers. For example, PyTorch's model.eval() does this work.

More about dropout:

| improve this answer | |
$\endgroup$
  • $\begingroup$ I did not know PyTorch automatically deactivated drop out layers $\endgroup$ – Tobi Mar 3 at 16:18
0
$\begingroup$

Keras turns off dropout when computing validation accuracy during training. Also when making predictions

| improve this answer | |
$\endgroup$
  • $\begingroup$ is this why my training accuracy is lower than validation/test? $\endgroup$ – Tobi Mar 3 at 17:37
  • $\begingroup$ No, it's not. Training loss and accuracy are usually greater then validation. This is because overfiting $\endgroup$ – Aray Karjauv Mar 4 at 8:37
  • 1
    $\begingroup$ Welcome to AI.SE @GerryP! On this site, we prefer answers that provide a little more detail. Please consider adding a few more sentences to your answer, and/or including some references. $\endgroup$ – John Doucette Mar 5 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.