Does each filter in each convolution layer create a new image? [duplicate]

Question

Say I have a CNN with this structure:

• input = 1 image (say, 30x30 RGB pixels)
• first convolution layer = 10 5x5 convolution filters
• second convolution layer = 5 3x3 convolution filters
• one dense layer with 1 output

So a graph of the network will look like this:

Am I correct in thinking that the first convolution layer will create 10 new images, i.e. each filter creates a new intermediary 30x30 image (or 26x26 if I crop the border pixels that cannot be fully convoluted).

Then the second convolution layer, is that supposed to apply the 5 filters on all of the 10 images from the previous layer? So that would result in a total of 50 images after the second convolution layer.

And then finally the last FC layer will take all data from these 50 images and somehow combine it into one output value (e.g. the probability that the original input image was a cat).

Or am I mistaken in how convolution layers are supposed to operate?

Also, how to deal with channels, in this case RGB? Can I consider this entire operation to be separate for all red, green and blue data? I.e. for one full RGB image, I essentially run the entire network three times, once for each color channel? Which would mean I'm also getting 3 output values.

Answer 1

You are partially correct. On CNNs the output shape per layer is defined by the amount of filters used, and the application of the filters (dilation, stride, padding, etc.).

CNNs shapes

In your example, your input is 30 x 30 x 3. Assuming stride of 1, no padding, and no dilation on the filter, you will get a spatial shape equal to your input, that is 30 x 30. Regarding the depth if you have 10 filters (of shape 5 x 5 x 3) you will end up with a 30 x 30 x 10 output at your first layer. Similarly, on the second layer with 5 filters (of shape 3 x 3 x 10, note the depth to work on the previous layer) you have 30 x 30 x 5 output. The FC layer has the same amount of weights as the input (that is 4500 weights) to create a linear combination of them.

CNN vs Convolution

Note that the CNNs operate differently from the traditional signal processing convolution. In the former, the convolution operation performs a dot product with the filter and the input to output a single value (and even add bias if you want to). While the latter outputs the same amount of channels.

The CNNs borrow the idea of a shifting kernel and a kernel response. But they do not apply a convolution operation per se.

Operation over the RGB

The CNN is not operating on each channel separately. It is merging the responses of the three channels and mixing them further. The deeper you get the more mix you get over your previous results.

The output of your FC is just one value. If you want more, you need to add more FC neurons to get more linear combinations of your inputs.

Answer 2

About the images inside the CNN layers: I really recommend this article since there is no one short answer to this question and it probably will be better to experiment with it.

About the RGB input images: When needed to train on RGB pictures it is not advised to split the RGB channels, you can think of it by trying to identify a fictional cat with red ears, green body and blue tail. Each separated channel don't represent a cat, most certainly not with high confidence. I would recommend to transform you RGB images to gray scale and measure the network performance. If the performance are not sufficient you can make a 3D convolution layer. For example: If 30x30x3 is the input image, the filter has to be NxNx3.

Answer 3

For a 3 channel image (RGB), each filter in a convolutional layer computes a feature map which is essentially a single channel image. Typically, 2D convolutional filters are used for multichannel images. This can be a single filter applied to each layer or a seperate filter per layer. These filters are looking for features which are independent of the color, i.e. edges (if you are looking for color there are far easier ways than CNNs). The filter is applied to each channel and the results are combined into a single output, the feature map. Since all channels are used by the filter to compute a single feature map, the number of channels in the input does not affect the structure of the network beyond the first layer. The size of a feature map is determined by the filter size, stride, padding and dilation(not commonly used - see here if you are interested.).

In your example, a 30 x 30 x 3 input convolved with 10 5 x 5 filters will yield a volume of 30 x 30 x 10 if the filters have a stride of 1 and same padding (or, 26 x 26 x 10 with valid padding / 34 x 34 x 10 with full padding).

Same padding buffers the edge of the input with filter_size/2 (integer division) to yield an output of equal size (assuming stride is 1) while valid padding would result in a smaller output. Valid padding doesn't crop the image as you said, it's more of a dilution of the signal at the edges, however the results is essentially the same. Note that even with same padding the edge pixels are used in fewer convolutions - a 5 x 5 filter with same padding will use a central pixel 25 times (every position on the filter) but only 9 times for a corner pixel. To use all pixels evenly full padding must be used which buffers the edge of the input with filter_size - 1.

                                          

Each feature map becomes a channel in the output volume. Therefore, the number of channels in the output volume is always equal to the number of filters in the convolutional layer. So, the second layer would output a volume of size 30 x 30 x 5 (stride 1, same padding).

The last layer in your example (fully connected) multiplies the value of each pixel in each feature map by a learned weight and sums the result. If the network is a binary classifier, the summed value results in a 1 or 0 output if a threshold is reached or as a decimal value for a regression model. This is determined by the FC neurons' activation function.

If visualizing this helps you as much as it helps me, I highly recommend having a look at the interactive examples here. Note that what is shown by this tool is the signal propagating through the network, i.e. the output from each layer, not the filters/weights themselves.

If you are interested in a bit more depth about ANNs and convolutional layers, I cover all the basics in my thesis(this is where the image is from) - p.9-16 ANNs & p.16-23 CNNs.