1
$\begingroup$

I know I've seen this somewhere before, but can't find it now. Say we have a neural network with a handful of layers, and we're applying dropout to each layer. As we move closer to the output, should dropout decrease, increase, or stay the same?

$\endgroup$
1
$\begingroup$

We can think of the dropout as of averaging over many networks. Not as a way to mute or denoise your data. Therefore the rule of thumb is the dropout should be applied to higher layers in your network. It is usually applied to fully connected layers (if at all!). It is a common practice to keep 0.5-0.75. neurons active. Personally I prefer L2 much more.

$\endgroup$
5
  • $\begingroup$ So dropout should decrease as we get closer to the output. Why do you prefer L2? $\endgroup$ – wordsforthewise Apr 15 '17 at 1:22
  • 1
    $\begingroup$ 1. It seems to be more stable in general. 2. When you have a large pretrained network and you want to keep the architecture unchanged and you want to use it to work with a smaller amount of data - I've empirically found that with little data you can apply strong L2 regularization with v. good effects (to fc layers - not to conv layers). The same doesn't apply to dropout. Keeping less than 0.5 neurons may result in v. unstable training. $\endgroup$ – FunkyKowal Apr 15 '17 at 12:37
  • $\begingroup$ OK thanks. Do you know if it's beneficial to have dropout on more than one layer, or is one usually good enough? $\endgroup$ – wordsforthewise Apr 15 '17 at 20:40
  • $\begingroup$ Related question: is it best to use L2 on just 1 layer or all fully_connected? $\endgroup$ – wordsforthewise Apr 15 '17 at 20:41
  • $\begingroup$ I don't think I've ever thought of applying l2 to only some of fc layers. Although, since you seem like looking for a general advice I would suggest getting the architecture right without focusing on the details too much, and getting to know your architecture through experimenting (on the large changes first, keep it simple). $\endgroup$ – FunkyKowal Apr 16 '17 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.