1
$\begingroup$

CNNs are naturally translation equivariant, meaning that if we translate the input, then the feature maps are translated the equally.

With the use of max/avg pooling layers, this translation equivariance leads to approximate translation invariance, in the sense that it gives translation invariance for small translations, but for longer translation, the max/avg values could differ due to the limitations by the size of the pooling and the module of the translation.

However, if I use the biggest size possible for the pooling, i.e. global max/avg pooling, then the CNN will be translation invariant in an exact way, for every translation of the input, no matter the strength (module) of the translation.

Is this intuition correct? I can visualize this in my head, but I can't really visualize if this effect would hold for deep networks (i.e. after many conv->global pool layers are stacked).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Your intuition is correct, when you apply many local feature detectors (convolutions) and throw the results into one big pot (any global pooling), only the amount of detected features matters, not their location in the image, i.e. you achieved translation invariance. Doing this for deep ConvNets like you describe does not make a lot of sense to me, because applying the global pooling once will squash your feature map into a single feature vector. When you look at the shape before and after the global pooling operation, this would look as follows:

[batch, height, width, channels] --global-pool--> [batch, channels]

Therefore your notion of space disappears completely which would prevent you from applying any subsequent spatial convolutions.

Improve this answer
$\endgroup$
4
  • $\begingroup$ amazing! this is exactly what I thought. So that's why usually, the global pooling layer is used only at the very last layer (in deep CNNs), and regular small sized poolings are used on the rest of the layers. Only thing is, these deep CNNs are then not exactly translation invariant, as the exact invariance will be lost in the small sized pooling layers. Is it then that we can not achieve exact translation equivariance for deep CNNs? (without augmentation or conveniently transforming data, only by architecture designs) $\endgroup$
    – puradrogasincortar
    Mar 15 at 15:39
  • 1
    $\begingroup$ That's exactly right! Regarding the second point: (I'm assuming you mixed up the terms invariance and equivariance a bit in your comment) Firstly, you are right, when local pooling is applied you trade local equivariance for local invariance and the more pooling layers you use the more this applies. However, it is possible to achieve full translational equivariance by using a fully convolutional model without pooling layers (often used in image segmentation tasks). It depends on the task how equivariant or invariant the model should be. I hope this clears it up a bit? $\endgroup$
    – Chillston
    Mar 15 at 16:11
  • 1
    $\begingroup$ oops yeah I meant invariance instead of equivariance in my last point. I understood everything! I imagine that fully convolutional models make use of padding and strides conveniently to decrease the spatial size at a higher rate; otherwise, it would take an enormous amount of conv layers to obtain a let's say 10 sized output from a high resolution image. Thanks, all clear! $\endgroup$
    – puradrogasincortar
    Mar 15 at 21:13
  • $\begingroup$ That's right! I'm glad it helped :) $\endgroup$
    – Chillston
    Mar 16 at 12:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .