3
$\begingroup$

If I got well the global idea of DropOut it allows to improve the sparsity of the information that come from one layer to another by setting some weights to zero.

In another hand, pooling, let's say max pooling, takes the maximum value in a neighborhood, reducing as well to zero, the influence of values apart from this maximum.

Without considering shape transformation due to pooling layer, can we say that pooling is a kind of DropOut step ?

Does adding DropOut or DropConnect layer after a pooling layer has a sense in CNN? And does it help further more the training process and generalization property ?

$\endgroup$
5
$\begingroup$

Dropout and Max-pooling are performed for different reasons.

Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout is reduce unnecessary feature dependencies in the network, allowing it to be simpler and improves its generalization abilities (reduces overfitting). In simple terms, it helps the model to learn that some features are an "OR" and not an "AND" requirements.

Max-pooling is not a regularization technique and it is part of the model's architecture, so it is also used during evaluation. The goal of max-pooling is to down-sample an input representation. As a result the model becomes less sensitive to some translations (improving translation invariance).

As for your last question, yes. dropout can be used after a pooling layer.

$\endgroup$
  • $\begingroup$ Clean answer thank you. If I understood well, features maps in CNN are linked through convolution and pooling layers. Where could we introduce the drop out ? (In dense connection weights are obvious, and so is the gain in sparsity) $\endgroup$ – nsaura Dec 17 '18 at 13:54
  • 1
    $\begingroup$ The dropout would be used after the pooling layers. I will say that it is very uncommon to use dropout in the convolution layers. In the convolution layers it can be viewed as adding occlusion, which helps the model to focus on smaller features. $\endgroup$ – Mark.F Dec 18 '18 at 9:20
1
$\begingroup$

I think we would consider regularization and downsampling better in this way:

  1. dropout

it puts some input value (neuron) for the next layer as 0, which makes the current layer a sparse one. So it reduces the dependence of each feature in this layer.

  1. pooling layer

the downsampling directly remove some input, and that makes the layer "smaller" rather than "sparser". The difference can be subtle but clear enough.

That's the root reason why the former also affect the evaluation/test process but the later does not.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.