5
$\begingroup$

If I got well the idea of dropout, it allows improving the sparsity of the information that comes from one layer to another by setting some weights to zero.

On the other hand, pooling, let's say max-pooling, takes the maximum value in a neighborhood, reducing as well to zero, the influence of values apart from this maximum.

Without considering the shape transformation due to the pooling layer, can we say that pooling is a kind of a dropout step?

Would the addition of a dropout (or DropConnect) layer, after a pooling layer, make sense in a CNN? And does it help the training process and generalization property?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
8
$\begingroup$

Dropout and Max-pooling are performed for different reasons.

Dropout is a regularization technique, which affects only the training process (during evaluation, it is not active). The goal of dropout is reduce unnecessary feature dependencies in the network, allowing it to be simpler and improves its generalization abilities (reduces overfitting). In simple terms, it helps the model to learn that some features are an "OR" and not an "AND" requirements.

Max-pooling is not a regularization technique and it is part of the model's architecture, so it is also used during evaluation. The goal of max-pooling is to down-sample an input representation. As a result the model becomes less sensitive to some translations (improving translation invariance).

As for your last question, yes. dropout can be used after a pooling layer.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Clean answer thank you. If I understood well, features maps in CNN are linked through convolution and pooling layers. Where could we introduce the drop out ? (In dense connection weights are obvious, and so is the gain in sparsity) $\endgroup$
    – nsaura
    Commented Dec 17, 2018 at 13:54
  • 1
    $\begingroup$ The dropout would be used after the pooling layers. I will say that it is very uncommon to use dropout in the convolution layers. In the convolution layers it can be viewed as adding occlusion, which helps the model to focus on smaller features. $\endgroup$
    – Mark.F
    Commented Dec 18, 2018 at 9:20
2
$\begingroup$

I think we would consider regularization and downsampling better in this way:

  1. dropout

it puts some input value (neuron) for the next layer as 0, which makes the current layer a sparse one. So it reduces the dependence of each feature in this layer.

  1. pooling layer

the downsampling directly remove some input, and that makes the layer "smaller" rather than "sparser". The difference can be subtle but clear enough.

That's the root reason why the former also affect the evaluation/test process but the later does not.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .