2
$\begingroup$

Does a fully convolutional network share the same translation invariance properties we get from networks that use max-pooling?

If not, why do they perform as well as networks which use max-pooling?

$\endgroup$
  • $\begingroup$ Convolution is translationally invariant $\endgroup$ – mirror2image Feb 22 at 8:35
  • $\begingroup$ It's not. The output of a convolutional layer will be shifted. $\endgroup$ – FourierFlux Feb 22 at 21:29
0
$\begingroup$

All convolutional networks (with or without max-pooling) are translation-invariant (AKA spatially invariant) because their filters slide over every position in the image. This means that if a pattern that "matches" a filter is present anywhere in the image, then at least one neuron should activate.

Max-pooling, on the other hand, has nothing to do with spatial invariance. It's simply a regularization technique to help reduce the number of parameters later in the network by downsizing activation layers within the network. This can help combat overfitting, although it's not strictly necessary. Alternatively, neural networks can achieve the same effect by using a convolutional layer with a stride of 2 instead of 1.

| improve this answer | |
$\endgroup$
  • $\begingroup$ what? Max pooling specifically creates translation invariance by only outputting the max value over a subset of the input space. Fully convolutional networks does not have this so I'm curious how/if it retains translation invariant properties. There are some architectures which have a bunch of Convolutional layers with strides to reduce output size with a softmax or SVM at the output. It's not clear why this is translation invariant at all. $\endgroup$ – FourierFlux Feb 21 at 1:04
  • $\begingroup$ Your description of max pooling is correct but that is not a significant form of spatial invariance. The convolutional layers themselves induce spatial invariance through the sliding action of their filters, so max pooling is not always needed. I think this article explains the difference well. Specifically, see the section called Getting rid of pooling. $\endgroup$ – Brett Daley Feb 21 at 1:57
  • 1
    $\begingroup$ The convolution output is shifted if the input is shifted, it's output is not translation invariant. Max pooling specifically acts to produce translation invariance(at least according to the books I have read). $\endgroup$ – FourierFlux Feb 21 at 1:59
  • $\begingroup$ That is true, but if you shift the input then the output of max pooling can change too. For example, imagine I shift my input image by 1 pixel when I am using 2x2 max pooling with a stride of 2. I am curious if you remember which book says that, so I can check to make sure I understand correctly. $\endgroup$ – Brett Daley Feb 21 at 2:57
  • $\begingroup$ Does this Quora answer help? Looks like the OP is referring to the Deep Learning Book. It appears that both convolutional layers and max pooling provide (slightly different) forms of translation invariance. That might explain why you can remove max pooling from CNNs and still have translation invariance, like I said in my answer. $\endgroup$ – Brett Daley Feb 21 at 3:03
0
$\begingroup$

FCNs can and typically have downsampling operations. For example, u-net has downsampling (more precisely, max-pooling) operations. The difference between an FCN and a regular CNN is that the former does not have fully connected layers. See this answer for more info.

Therefore, FCNs inherit the same properties of CNNs. There's nothing that a CNN (with fully connected layers) can do that an FCN cannot do. In fact, you can even simulate a fully connected layer with a convolution (with a kernel that has the same shape as the input volume).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.