What happens to the channels after the convolution layer?

I wonder what happens to the 'channels' dimension (usually 3 for RGB images) after the first convolution layer in CNNs?

In books and other sources, it is always said that the depth of the output from convolutional layers is the number of kernels (filters) in that layer.

But, if the input image has 3 channels and we convolve each of them with $$K$$ kernels, shouldn't the depth of the output be $$K * 3$$? Are they somehow 'averaged' or in other way combined with each other?

• This question has been asked multiple times in the past. Your filters will typically have a depth equal to the depth of the input. Does this ai.stackexchange.com/q/17783/2444 or ai.stackexchange.com/a/18669/2444 answer your question? Maybe have a look at ai.stackexchange.com/q/3287/2444 or ai.stackexchange.com/q/5769/2444 too. Tell me which one answers your question, then I will mark this as a duplicate of that one.
– nbro
Apr 27 '20 at 12:26
• I would say that none of these question is excactly what I meant BUT answer to my question is somewhere among answers to this one: ai.stackexchange.com/q/5769/22659. The direct answer to my question is that values obtained from convolutions among different channels sum up together, therefore 3 channels after convolution with one filter give one output. The best explanation delivered by Andrew: coursera.org/lecture/convolutional-neural-networks/… Apr 27 '20 at 15:08
• I think you shouldn't think of kernels (or filters) as 2d matrices, but as (3d) tensors. Anyway, feel free to provide an answer to your own question (if this isn't really a duplicate)! Also, don't forget next time to tag me with @nbro, otherwise, I don't see your comments.
– nbro
Apr 28 '20 at 14:28

• I think this answer is a little bit misleading. What happens is that the kernel actually has the same depth as the depth of the input. So, if the input has 3 channels, then a $3 \times 3$ kernel is actually a $3 \times 3 \times 3$ kernel. From your explanation, it seems like you're suggesting that you perform different convolutions (with the same or different kernel?) for each channel, then sum up the results. That's somehow equivalent, but, as I said, in my opinion, conceptually more complicated than thinking that the kernel has the same depth as the input.