# How can 3 same size CNN layers in different ordering output different receptive field from the input layer?

Below is a quote from CS231n:

Prefer a stack of small filter CONV to one large receptive field CONV layer. Suppose that you stack three 3x3 CONV layers on top of each other (with non-linearities in between, of course). In this arrangement, each neuron on the first CONV layer has a 3x3 view of the input volume. A neuron on the second CONV layer has a 3x3 view of the first CONV layer, and hence by extension a 5x5 view of the input volume. Similarly, a neuron on the third CONV layer has a 3x3 view of the 2nd CONV layer, and hence a 7x7 view of the input volume. Suppose that instead of these three layers of 3x3 CONV, we only wanted to use a single CONV layer with 7x7 receptive fields. These neurons would have a receptive field size of the input volume that is identical in spatial extent (7x7), but with several disadvantages

My visualized interpretation: How can you see through the first CNN layer from the second CNN layer and see a 5x5 sized receptive field?

There were no previous comments stating all the other hyperparameters, like input size, steps, padding, etc. which made this very confusing to visualize.

Edited:

I think I found the answer. BUT I still don't understand it. In fact, I am more confused than ever.

It is really easy to visualize the growth in the receptive field of the input as you go deep into the CNN layers if you consider a small example.

Let's take a simple example:

The dimensions are in the form of $$\text{channels} \times \text{height} \times \text{width}$$.

• The input image $$I$$ is a $$3 \times 5 \times 5$$ matrix
• The first convolutional layer's kernel $$K_1$$ has shape $$3 \times 2 \times 2$$ (we consider only 1 filter for simplicity)
• The second convolutional layer's kernel $$K_2$$ has shape $$1 \times 2 \times 2$$
• Padding $$P = 0$$
• Stride $$S = 1$$

The output dimensions $$O$$ are calculated by the following formula taken from the lecture CS231n.

$$O= (I - K + 2P)/S + 1$$

When you do a convolution of the input image with the first filter $$K_1$$, you get an output of shape $$1 \times 4 \times 4$$ (this is the activation of the CONV1 layer). The receptive field for this is the same as the kernel size ($$K_1$$), that is, $$2 \times 2$$.

When this layer (of shape $$1 \times 4 \times 4$$) is convolved with the second kernel (CONV2) $$K_2$$ ($$1 \times 2 \times 2$$), the output would be $$1 \times 3 \times 3$$. The receptive field for this would be the $$3 \times 3$$ window of the input because you have already accumulated the sum of the $$2 \times 2$$ window in the previous layer.

Considering your example of three CONV layers with $$3 \times 3$$ kernels is also similar. The first layer activation accumulates the sum of all the neurons in the $$3 \times 3$$ window of the input. When you further convolve this output with a kernel of $$3 \times 3$$, it will accumulate all the outputs of the previous layers covering a bigger receptive field of the input.

This observation comes in line with the argument that deeper layers learn more intricate features like facial expressions, abstract concepts, etc. because they cover a larger receptive field of our original input image.

The problem is in your diagram. Here are the steps to get to a 5x5 receptive field. Here is your diagram, redone slightly: Notice that the new unit takes a weighted sum of the 9 pixels in the input, and then applies a rectified linear nonlinearity. Now, there are more of these, creating three new numbers computed from that part of the image. Each one slides over by one pixel: We repeat this process going down three pixels as well, and then finally, we have a new 3x3 input field: Notice that the new unit on the right now gets input from a 5x5 input field. I hope this helps!

The intention of the referred text is to reason out the disadvantage of equivalent-merged-single-convolution-layer over multiple [CONV -> RELU]*N layers.

In the given scenario, if 2 layers of 3x3 filters were to be replaced by an equivalent single layer then this equivalent layer would need a filter with a receptive field of size 5x5.

Similarly, an equivalent layer filter would need its receptive field to be of size 7x7 to compress 3 layers of 3x3 filters. Note that the most obvious disadvantage would be missing out on modeling non-linearity.