Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?

If not, why do they perform as well as networks which use max-pooling?

  • $\begingroup$ Convolution is translationally invariant $\endgroup$ – mirror2image Feb 22 '20 at 8:35
  • $\begingroup$ It's not. The output of a convolutional layer will be shifted. $\endgroup$ – FourierFlux Feb 22 '20 at 21:29

Neural networks are not invariant to translations, but equivariant,

Invariance vs Equivariance

Suppose we have input $x$ and the output $y=f(x)$ of some map between spaces $X$ and $Y$. We apply transformation $T$ in the input domain. For general map,output will change in some complicated and unpredictable way. However, for certain class of maps, change of the output becomes very tractable.

Invariance means that output doesn't change after application of the map $T$. Namely: $$ f(T(x)) = f(x) $$

For CNN example of the map, invariant to translations, is the GlobalPooling operation.

Equivariance means that symmetry transformation $T$ on the input domain leads to the symmetry transformation $T^{'}$ on the output. Here $T^{'}$ can be the same map $T$, identity map - which reduces to invariance, or some other kind of transformation.

This picture is illustration of translational equivariance.

from https://towardsdatascience.com/translational-invariance-vs-translational-equivariance-f9fbc8fca63a

Equivariance of operations in CNN

  • Convolutions with stride=1: $$ f(T(x)) = T f(x) $$ Output feature map is shifted in same direction and number of steps.
  • Downsampling operations. Convolutions with stride=1, Pooling (non-global): $$ f(T_{1/s}(x)) = T_{1/s} f(x) $$ They are equivariant to the subgroup of translations, which involves translations with integer number of strides.
  • GlobalPooling : $$ f(T(x)) = f(x) $$ These are invariant to arbitrary shifts, this property is useful in classification tasks.

Combination of layers

Stacking multiple equivariant layers you obtain equivariant architecture a whole.

For classification layer it makes sense to put GlobalPooling in the end in order to for NN to output the same probabilities for the shifted image.

For segmentation or detection problem architecture should be equivariant with the same map $T$, in order to translate bounding boxes or segmentation masks by the same amount as the transform on the input.

Non-global downsampling operations reduce equivariance to the subgroup with shifts integer multiples of stride.


All convolutional networks (with or without max-pooling) are translation-invariant (AKA spatially invariant) because their filters slide over every position in the image. This means that if a pattern that "matches" a filter is present anywhere in the image, then at least one neuron should activate.

Max-pooling, on the other hand, has nothing to do with spatial invariance. It's simply a regularization technique to help reduce the number of parameters later in the network by downsizing activation layers within the network. This can help combat overfitting, although it's not strictly necessary. Alternatively, neural networks can achieve the same effect by using a convolutional layer with a stride of 2 instead of 1.

  • $\begingroup$ what? Max pooling specifically creates translation invariance by only outputting the max value over a subset of the input space. Fully convolutional networks does not have this so I'm curious how/if it retains translation invariant properties. There are some architectures which have a bunch of Convolutional layers with strides to reduce output size with a softmax or SVM at the output. It's not clear why this is translation invariant at all. $\endgroup$ – FourierFlux Feb 21 '20 at 1:04
  • $\begingroup$ Your description of max pooling is correct but that is not a significant form of spatial invariance. The convolutional layers themselves induce spatial invariance through the sliding action of their filters, so max pooling is not always needed. I think this article explains the difference well. Specifically, see the section called Getting rid of pooling. $\endgroup$ – Brett Daley Feb 21 '20 at 1:57
  • 1
    $\begingroup$ The convolution output is shifted if the input is shifted, it's output is not translation invariant. Max pooling specifically acts to produce translation invariance(at least according to the books I have read). $\endgroup$ – FourierFlux Feb 21 '20 at 1:59
  • $\begingroup$ That is true, but if you shift the input then the output of max pooling can change too. For example, imagine I shift my input image by 1 pixel when I am using 2x2 max pooling with a stride of 2. I am curious if you remember which book says that, so I can check to make sure I understand correctly. $\endgroup$ – Brett Daley Feb 21 '20 at 2:57
  • $\begingroup$ Does this Quora answer help? Looks like the OP is referring to the Deep Learning Book. It appears that both convolutional layers and max pooling provide (slightly different) forms of translation invariance. That might explain why you can remove max pooling from CNNs and still have translation invariance, like I said in my answer. $\endgroup$ – Brett Daley Feb 21 '20 at 3:03

FCNs can and typically have downsampling operations. For example, u-net has downsampling (more precisely, max-pooling) operations. The difference between an FCN and a regular CNN is that the former does not have fully connected layers. See this answer for more info.

Therefore, FCNs inherit the same properties of CNNs. There's nothing that a CNN (with fully connected layers) can do that an FCN cannot do. In fact, you can even simulate a fully connected layer with a convolution (with a kernel that has the same shape as the input volume).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.