Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?

Question

Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?

If not, why do they perform as well as networks which use max-pooling?

Answer 1

Neural networks are not invariant to translations, but equivariant,

Invariance vs Equivariance

Suppose we have input $x$ and the output $y=f(x)$ of some map between spaces $X$ and $Y$. We apply transformation $T$ in the input domain. For general map,output will change in some complicated and unpredictable way. However, for certain class of maps, change of the output becomes very tractable.

Invariance means that output doesn't change after application of the map $T$. Namely: $$ f(T(x)) = f(x) $$

For CNN example of the map, invariant to translations, is the GlobalPooling operation.

Equivariance means that symmetry transformation $T$ on the input domain leads to the symmetry transformation $T^{'}$ on the output. Here $T^{'}$ can be the same map $T$, identity map - which reduces to invariance, or some other kind of transformation.

This picture is illustration of translational equivariance.

Equivariance of operations in CNN

• Convolutions with stride=1: $$ f(T(x)) = T f(x) $$ Output feature map is shifted in same direction and number of steps.
• Downsampling operations. Convolutions with stride=1, Pooling (non-global): $$ f(T_{1/s}(x)) = T_{1/s} f(x) $$ They are equivariant to the subgroup of translations, which involves translations with integer number of strides.
• GlobalPooling : $$ f(T(x)) = f(x) $$ These are invariant to arbitrary shifts, this property is useful in classification tasks.

Combination of layers

Stacking multiple equivariant layers you obtain equivariant architecture a whole.

For classification layer it makes sense to put GlobalPooling in the end in order to for NN to output the same probabilities for the shifted image.

For segmentation or detection problem architecture should be equivariant with the same map $T$, in order to translate bounding boxes or segmentation masks by the same amount as the transform on the input.

Non-global downsampling operations reduce equivariance to the subgroup with shifts integer multiples of stride.

Answer 2

FCNs can and typically have downsampling operations. For example, u-net has downsampling (more precisely, max-pooling) operations. The difference between an FCN and a regular CNN is that the former does not have fully connected layers. See this answer for more info.

Therefore, FCNs inherit the same properties of CNNs. There's nothing that a CNN (with fully connected layers) can do that an FCN cannot do. In fact, you can even simulate a fully connected layer with a convolution (with a kernel that has the same shape as the input volume).

Answer 3

All convolutional networks (with or without max-pooling) are translation-invariant (AKA spatially invariant) because their filters slide over every position in the image. This means that if a pattern that "matches" a filter is present anywhere in the image, then at least one neuron should activate.

Max-pooling, on the other hand, has nothing to do with spatial invariance. It's simply a regularization technique to help reduce the number of parameters later in the network by downsizing activation layers within the network. This can help combat overfitting, although it's not strictly necessary. Alternatively, neural networks can achieve the same effect by using a convolutional layer with a stride of 2 instead of 1.