How can 3 same size CNN layers in different ordering output different receptive field from the input layer?

Question

Below is a quote from CS231n:

Prefer a stack of small filter CONV to one large receptive field CONV layer. Suppose that you stack three 3x3 CONV layers on top of each other (with non-linearities in between, of course). In this arrangement, each neuron on the first CONV layer has a 3x3 view of the input volume. A neuron on the second CONV layer has a 3x3 view of the first CONV layer, and hence by extension a 5x5 view of the input volume. Similarly, a neuron on the third CONV layer has a 3x3 view of the 2nd CONV layer, and hence a 7x7 view of the input volume. Suppose that instead of these three layers of 3x3 CONV, we only wanted to use a single CONV layer with 7x7 receptive fields. These neurons would have a receptive field size of the input volume that is identical in spatial extent (7x7), but with several disadvantages

My visualized interpretation:

How can you see through the first CNN layer from the second CNN layer and see a 5x5 sized receptive field?

There were no previous comments stating all the other hyperparameters, like input size, steps, padding, etc. which made this very confusing to visualize.

---

Edited:

I think I found the answer. BUT I still don't understand it. In fact, I am more confused than ever.

Answer 1

It is really easy to visualize the growth in the receptive field of the input as you go deep into the CNN layers if you consider a small example.

Let's take a simple example:

The dimensions are in the form of $\text{channels} \times \text{height} \times \text{width}$.

• The input image $I$ is a $3 \times 5 \times 5$ matrix
• The first convolutional layer's kernel $K_1$ has shape $3 \times 2 \times 2$ (we consider only 1 filter for simplicity)
• The second convolutional layer's kernel $K_2$ has shape $1 \times 2 \times 2$
• Padding $P = 0$
• Stride $S = 1$

The output dimensions $O$ are calculated by the following formula taken from the lecture CS231n.

$$O= (I - K + 2P)/S + 1$$

When you do a convolution of the input image with the first filter $K_1$, you get an output of shape $1 \times 4 \times 4$ (this is the activation of the CONV1 layer). The receptive field for this is the same as the kernel size ($K_1$), that is, $2 \times 2$.

When this layer (of shape $1 \times 4 \times 4$) is convolved with the second kernel (CONV2) $K_2$ ($1 \times 2 \times 2$), the output would be $1 \times 3 \times 3$. The receptive field for this would be the $3 \times 3$ window of the input because you have already accumulated the sum of the $2 \times 2$ window in the previous layer.

Considering your example of three CONV layers with $3 \times 3$ kernels is also similar. The first layer activation accumulates the sum of all the neurons in the $3 \times 3$ window of the input. When you further convolve this output with a kernel of $3 \times 3$, it will accumulate all the outputs of the previous layers covering a bigger receptive field of the input.

This observation comes in line with the argument that deeper layers learn more intricate features like facial expressions, abstract concepts, etc. because they cover a larger receptive field of our original input image.

Answer 2

The problem is in your diagram. Here are the steps to get to a 5x5 receptive field. Here is your diagram, redone slightly:

Notice that the new unit takes a weighted sum of the 9 pixels in the input, and then applies a rectified linear nonlinearity. Now, there are more of these, creating three new numbers computed from that part of the image. Each one slides over by one pixel:

We repeat this process going down three pixels as well, and then finally, we have a new 3x3 input field:

Notice that the new unit on the right now gets input from a 5x5 input field. I hope this helps!

Answer 3

The intention of the referred text is to reason out the disadvantage of equivalent-merged-single-convolution-layer over multiple [CONV -> RELU]*N layers.

In the given scenario, if 2 layers of 3x3 filters were to be replaced by an equivalent single layer then this equivalent layer would need a filter with a receptive field of size 5x5.

Similarly, an equivalent layer filter would need its receptive field to be of size 7x7 to compress 3 layers of 3x3 filters. Note that the most obvious disadvantage would be missing out on modeling non-linearity.