How can I suppress a CNN’s translation invariant or translation equivariant?

Question

I am trying to understand this post, but I get confused by the definitions and the differences. What's definition of equivariant?

If I remove all the pooling layers from a CNN, will it make the network to detect features in pixel resolution? For example, detecting the local maximum of a pixel. For example, can a CNN be designed to return True for the following case?

And False for the shifted window:

In the second case it returns false because the 3x3 submatrix isn't centered (yellow dash line) around the local maximum.

Will an architecture that is

from keras.layers import Dense, Conv2D, Flatten
model = Sequential()
model.add(Conv2D(128, kernel_size=2, activation=’relu’, padding='same', input_shape=(3,3,1)))
model.add(Conv2D(64, kernel_size=2, activation=’relu’, padding='same'))
model.add(Flatten())
model.add(Dense(10, activation=’softmax’))

be able to differentiate between the tiling of the larger grayscale image?

Answer 1

Question 1

To make it simple

• You have a transformation $T$ and an operator $C$ acting on a given input $x$

• Let's say you do this experiment

  • compute $y_{1} = C(T(x))$
  • compute $y_{2} = C(x)$

• You can get three different results:

  • $y_{1} = y_{2}$ then you can say the operator is invariant with respect to the given transformation
  • $y_{1} = T(y_{2})$ then you can say the operator is equivariant to the given transformation as applying it to the input basically reflects its effect completely on the output
  • none of the 2

Questions 2

Q2.1

• Spatial Pooling is not the only way to perform dimensionality reduction. You can achieve it even simply applying the Conv Kernel with no padding.

For example, let's say your input is a WxHxC tensor and you are applying a kernel which is KxKxC the result will have spatial domain size (W-(K/2))x(H-(K/2)) with K/2 the integer truncation so if K=5 then K/2=2.

Alternatively you can reduce the spatial domain with stride.

Q2.2

• It seems to me you are talking about a sort of Non Max Suppression Operator rather an emergent behaviour, but certainly with the proper supervision signal you can train a CNN to do this work (even if practically it does not make sense as you can explicitly define it).