# How can 3 same size CNN layers in different ordering output different receptive field from the input layer?

Below is a quote from CS231n

Prefer a stack of small filter CONV to one large receptive field CONV layer. Suppose that you stack three 3x3 CONV layers on top of each other (with non-linearities in between, of course). In this arrangement, each neuron on the first CONV layer has a 3x3 view of the input volume. A neuron on the second CONV layer has a 3x3 view of the first CONV layer, and hence by extension a 5x5 view of the input volume. Similarly, a neuron on the third CONV layer has a 3x3 view of the 2nd CONV layer, and hence a 7x7 view of the input volume. Suppose that instead of these three layers of 3x3 CONV, we only wanted to use a single CONV layer with 7x7 receptive fields. These neurons would have a receptive field size of the input volume that is identical in spatial extent (7x7), but with several disadvantages.

My visualized interpretation How can you see through the first CNN layer from the second CNN layer and see a 5x5 sized receptive field? There were no previous comments stating all the other hyperparameters, like input size, steps, padding, which made this very confusing to visualize.

Edited:

BUT I still don't understand it. In fact, I am more confused than ever.

It is really easy to visualize the growth in the receptive field of the input as you go deep into the CNN layers if you consider a small example.

Lets take a simple example :

Dimensions are in the form of (channels x height x width)

Input Image - 3x5x5 (I)

Conv 1 kernel - 3x2x2 (K1) : we consider only 1 filter

Conv 2 kernel - 1x2x2 (K2)

Stride = 1 (S)

The output dimensions are calculated by the following formula taken from the lecture CS231n.

Output = (I - K + 2P)/S + 1

When you do a convolution of the input image with the first filter K1, you get an output of 1x4x4 (this is the activation of CONV1 layer). The receptive field for this is same as the kernel size (K1) that is 2x2. When this layer (1x4x4) is convolved with the second kernel (CONV2) 1x2x2 (K2), the output would be 1x3x3. The receptive field for this would be 3x3 window of the input because you have already accumulated the sum of the 2x2 window in the previous layer.

Considering your example of three CONV layers with 3x3 kernels is also similar. The first layer activation accumulates the sum of all the neurons in the 3x3 window of the input. When you further convolve this output with a kernel of 3x3, it will accumulate all the outputs of the previous layers covering a bigger receptive field of the input.

This observation comes in line with the argument that deeper layers learn more intricate features like facial expressions, abstract concepts etc because they cover a larger receptive field of our original input image.

Hope this clarifies.

The problem is in your diagram. Here are the steps to get to a 5x5 receptive field. Here is your diagram, redone slightly: Notice that the new unit takes a weighted sum of the 9 pixels in the input, and then applies a rectified linear nonlinearity. Now, there are more of these, creating three new numbers computed from that part of the image. Each one slides over by one pixel: We repeat this process going down three pixels as well, and then finally, we have a new 3x3 input field: Notice that the new unit on the right now gets input from a 5x5 input field. I hope this helps!

The intention of the referred text is to reason out the disadvantage of equivalent-merged-single-convolution-layer over multiple [CONV -> RELU]*N layers.

In the given scenario, if 2 layers of 3x3 filters were to be replaced by an equivalent single layer then this equivalent layer would need a filter with Receptive Field of size 5x5.

Similarly, equivalent layer filter would need its Receptive Field to be of size 7x7 to compress 3 layers of 3x3 filters. NOTE that the most obvious disadvantage would be missing out on modeling non-linearity.