Are feature maps merged or are they passed on as they are?

Question

I am unsure about the following parts of the architecture and mechanics of convolution layers in CNNs. Possibly, this is implementation-dependent though.

First question:

Say I have 2 convolution layers with 10 filters each and the dimension of my input tensors is $n \times m \times 1$ (so, grayscale images for example). Passing this input to the first convolution layer results in 10 feature maps (10 matrices of $n \times m$, if we use padding), each produced by a different filter.

Now, what does actually happen when this is passed to the second convolution layer? Are all 10 feature maps passed as one big $m \times n \times 10$ tensor or are the overlapping cells of the 10 feature maps averaged and a $m \times n \times 1$ tensor is passed to the next convolution layer? The former would result in an explosion of feature maps with increasing number of convolution layers and the spacial complexity would be in $\mathcal{O}\left((nm)^k\right)$, where $k$ is the number of chained convolution layers. Averaging the feature maps before passing them to the next layer would keep the complexity linear. So, which is it? Or are both possibilities commonly used?

Second question (with two sub questions):

a) This is a similar question. If I have an input volume of $n \times m \times 3$ (e.g. RGB images) and I have again 2 convolution layers with 10 filters, does each convolution layer have in actuality 30 filters? So 10 sets of 3 filters, one for each channel? Or do I have in fact only 10 filters and the filters are applied to all 3 channels?

b) This is the same question as question (1) but for channels: Once I have convolved a filter (consisting of three channel filters? (a)) over the input tensor I end up with 3 feature maps. One for each channel. What do I do with these? Do I average them component-wise with each other? Or do I keep them separate until I have convolved all 10 filters across the input and THEN average the 10 feature maps of each channel? Or do I average all 30 feature maps of all three channels? Or do I just pass on 30 feature maps to the next convoloution layers which in turn knows which of these feature maps belong to which channel?

Quite a few possibilities... None of the sources I consulted makes this explicit. Maybe because it depends on the individual implementation.

Anyway, would be great if somebody could clear this confusion up a little!

Answer 1

tl;dr

It helps to think that the channels dimension of a convolutional layer works like a fully connected layer (i.e. the layer computes the weighted sum over all channels).

For a single pixel...

Let's consider a single pixel (e.g. the top left pixel). This pixel has $C$ different values, where $C$ are the number of channels. In order to produce the result of a single filter, the layer takes the weighted sum of these $C$ pixels. It does this by actually having $C$ weights, multiplying the pixel values with their corresponding weights and summing them together.

Example

Let's say you have an $n \times m \times 10$ tensor as an input to a convolutional layer with $1$ filter and a $3 \times 3$ kernel. For creating its output the layer has $3 \times 3 \times 10 = 90$ weights, i.e. a different $3 \times 3$ kernel for each of the $10$ input channels. To create its output, the layer performs the convolution operation separately on each of the input channels (each with its corresponding weight matrix) and creates this way $10$ feature maps which are summed together.

Now imagine the layer has $20$ filters instead of $1$. Nothing changes, just that the same procedure is done $20$ times with $20$ different sets of weights So the total number of weights in the layer in this case is $3 \times 3 \times 10 \times 20 = 1800$.

To answer your questions...

(1) You have a grayscale image $n \times m \times 1$ and it passes through the first convolution layer (which has $10$ filters). This layer will perform the convolution operation on the input image $10$ times independently and produce $10$ feature maps, i.e. an output tensor of $n \times m \times 10$. Now this in fed into the second convolution layer, which again has $10$ filters. For each filter the layer will perform the convolution operation on the $10$ input feature maps independently and will sum the corresponding pixels together to form a $n \times m \times 1$ feature map. It will perform this procedure $10$ times and generate an output tensor of $n \times m \times 10$.

(2) Yes, each layer actually has $30$ filters, a total of $10$ for each of the R, G, B channels. The layer just sums the results of the R, G, B channels to produce a single feature map. For the second part, if I got it right it's pretty much what you said but it sums the maps instead of averaging them

Notes

I'd recommend checking Stanford's CS231 notes on convolution layers, which explain it in much detail and also have numerical examples to confirm if you've got it right.

You could also check this answer for more details.

Answer 2

Answers:

1. Generally its the former. The next layer would learn at each filter how to merge the channels of the previous layer, that is why in a 2D convolution the kernel is a 3-dimensional tensor. But the number of parameters is $nmc_ic_{i+1}$ at the $i^{th}$ layer (this is ignoring bias). lets assume all channels are $O(c)$ then the spatial complexity becomes $O(knmc^2)$ where $k$ is the number of layers.
2. a) The first convolutional filter would have kernel size of (w,h,3,10) where w and h are the kernel sizes of the 2d convolution (often in practice is 3). So there are 10 filters of size (w,h,3), but the # of parameters is w*h*30 (once again for ease, ignoring bias). The second layer though, since it is working on a layer with 10 channels, will have kernel (w,h,10,10).
   b) I think you need to go back and look at what a convolution does (in your setting specifically a 2D convolution). Each filter works on every channel of the previous layer. Each channel of the last convolutional layer refers to a single filter that convolved over the entire previous layer to that.