In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

Question

My understanding is that the convolutional layer of a convolutional neural network has four dimensions: input_channels, filter_height, filter_width, number_of_filters. Furthermore, it is my understanding that each new filter just gets convoluted over ALL of the input_channels (or feature/activation maps from the previous layer).

HOWEVER, the graphic below from CS231 shows each filter (in red) being applied to a SINGLE CHANNEL, rather than the same filter being used across channels. This seems to indicate that there is a separate filter for EACH channel (in this case I'm assuming they're the three color channels of an input image, but the same would apply for all input channels).

This is confusing - is there a different unique filter for each input channel?

This is the source.

The above image seems contradictory to an excerpt from O'reilly's "Fundamentals of Deep Learning":

...filters don't just operate on a single feature map. They operate on the entire volume of feature maps that have been generated at a particular layer...As a result, feature maps must be able to operate over volumes, not just areas

...Also, it is my understanding that these images below are indicating a THE SAME filter is just convolved over all three input channels (contradictory to what's shown in the CS231 graphic above):

Answer 1

The following picture that you used in your question, very accurately describes what is happening. Remember that each element of the 3D filter (grey cube) is made up of a different value (3x3x3=27 values). So, three different 2D filters of size 3x3 can be concatenated to form this one 3D filter of size 3x3x3.

The 3x3x3 RGB chunk from the picture is multiplied elementwise by a 3D filter (shown as grey). In this case, the filter has 3x3x3=27 weights. When these weights are multiplied element-wise and then summed, it gives one value.

So, is there a separate filter for each input channel?

YES, there are as many 2D filters as the number of input channels in the image. However, it helps if you think that for input matrices with more than one channel, there is only one 3D filter (as shown in the image above).

Then why is this called 2D convolution (if the filter is 3D and the input matrix is 3D)?

This is 2D convolution because the strides of the filter are along the height and width dimensions only (NOT depth) and therefore, the output produced by this convolution is also a 2D matrix. The number of movement directions of the filter determines the dimensions of convolution.

Note: If you build up your understanding by visualizing a single 3D filter instead of multiple 2D filters (one for each layer), then you will have an easy time understanding advanced CNN architectures like Resnet, InceptionV3, etc.

Answer 2

In a convolutional neural network, is there a unique filter for each input channel or are the same new filters used across all input channels?

The former. In fact there is a separate kernel defined for each input channel / output channel combination.

Typically for a CNN architecture, in a single filter as described by your number_of_filters parameter, there is one 2D kernel per input channel. There are input_channels * number_of_filters sets of weights, each of which describe a convolution kernel. So the diagrams showing one set of weights per input channel for each filter are correct. The first diagram also shows clearly that the results of applying those kernels are combined by summing them up and adding bias for each output channel.

This can also be viewed as using a 3D convolution for each output channel, that happens to have the same depth as the input. Which is what your second diagram is showing, and also what many libraries will do internally. Mathematically this is the same result (provided the depths match exactly), although the layer type is typically labelled as "Conv2D" or similar. Similarly if your input type is inherently 3D, such as voxels or a video, then you might use a "Conv3D" layer, but internally it could well be implemented as a 4D convolution.

Answer 3

I'm following up on the answers above with a concrete example in the hope to further clarify how the convolution works with respect to the input and output channels and the weights, respectively:

Let the example be as follows (wrt to 1 convolutional layer):

• the input tensor is 9x9x5, i.e. 5 input channels, so input_channels=5
• the filter/kernel size is 4x4 and the stride is 1
• the output tensor is 6x6x56, i.e. 56 output channels, so output_channels=56
• the padding type is 'VALID' (i.e. no padding)

We note that:

• since the input has 5 channels, the filter dimension becomes 4x4x5, i.e. there are 5 separate, unique 2D filters of size 4x4 (i.e. each has 16 weights); in order to convolve over the input of size 9x9x5 the filter becomes 3D and must be of size 4x4x5
• therefore: for each input channel, there exists a distinct 2D filter with 16 different weights each. In other words, the number of 2D filters matches the number of input channels
• since there are 56 output channels, there must be 56 3-dimensional filters W0, W1, ..., W55 of size 4x4x5 (cf. in the CS231 graphic there are 2 3-dimensional filters W0, W1 to account for the 2 output channels), where the 3rd dimension of size 5 represents the link to the 5 input channels (cf. in the CS231 graphic each 3D filter W0, W1 has the 3rd dimension 3, which matches the 3 input channels)
• therefore: the number of 3D filters equals the number of output channels

That convolutional layer thus contains:

56 3-dimensional filters of size 4x4x5 (= 80 different weights each) to account for the 56 output channels where each has a value for the 3rd dimension of 5 to match the 5 input channels. In total there are

number_of_filters=input_channel*output_channels=5*56=280

2D filters of size 4x4 (i.e. 280x16 different weights in total).

Answer 4

I recommend chapter 2.2.1 of my masters thesis as an answer. To add to the remaining answers:

Keras is your friend to understand what happens:

from keras.models import Sequential
from keras.layers import Conv2D

model = Sequential()
model.add(Conv2D(32, input_shape=(28, 28, 3),
          kernel_size=(5, 5),
          padding='same',
          use_bias=False))
model.add(Conv2D(17, (3, 3), padding='same', use_bias=False))
model.add(Conv2D(13, (3, 3), padding='same', use_bias=False))
model.add(Conv2D(7, (3, 3), padding='same', use_bias=False))
model.compile(loss='categorical_crossentropy', optimizer='adam')

print(model.summary())

gives

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 28, 28, 32)        2400      
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 28, 28, 17)        4896      
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 28, 28, 13)        1989      
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 28, 28, 7)         819       
=================================================================
Total params: 10,104

Try to formulate your options. What would that mean for the parameters if something else would be the case?

Hint: $2400 = 32 \cdot (3 \cdot 5 \cdot 5)$

This approach also helps you with other layer types, not only convolutional layers.

Please also note that you are free to implement different solutions, that might have other numbers of parameters.

Answer 5

One addition to Mohsin's answer: try the awesome CNN explainer. There you can explore CNN's graphically, which makes things "click" very fast.

If you click on conv_1_1, you will see how three different kernels are used to calculate one convolution. That means you have one 2D filter per channel, that could together be interpreted as one 3D filter.

Answer 6

Just to make two details absolutely clear:

Say you have $N$ 2D input channels going to $N$ 2D output channels. The total number of 2D $3\times3$ filter weights is actually $N^2$. But how is the 3D convolution affected, i.e., if every input channel contributes one 2D layer to every output channel, then each output channel is composed initially of $N$ 2D layers, how are they combined?

This tends to be glossed over in almost every publication I've seen, but the key concept is the $N^2$ 2D output channels are interleaved with each other to form the $N$ output channels, like shuffled card decks, before being summed together. This is all logical when you realize that along the channel dimensions of a convolution (which is never illustrated), you actually have a fully connected layer! Every input 2D channel, multiplied by a unique $3\times 3$ filter, yields a 2D output layer contribution to a single output channel. Once combined, every output layer is a combination of every input layer $\times$ a unique filter. It's an all to all contribution.

The easiest way to convince yourself of this is to imagine what happens in other scenarios and see that the computation becomes degenerate - that is, if you don't interleave and recombine the results, then the different outputs wouldn't actually do anything - they'd have the same effect as a single output with combined weights.

Answer 7

For anyone trying to understand how convolutions are calculated, here is a useful code snippet in Pytorch:

batch_size = 1
height = 3 
width = 3
conv1_in_channels = 2
conv1_out_channels = 2
conv2_out_channels = 2
kernel_size = 2
# (N, C_in, H, W) is shape of all tensors. (batch_size, channels, height, width)
input = torch.Tensor(np.arange(0, batch_size*height*width*in_channels).reshape(batch_size, in_channels, height, width))
conv1 = nn.Conv2d(in_channels, conv1_out_channels, kernel_size, bias=False) # no bias to make calculations easier
# set the weights of the convolutions to make the convolutions easier to follow
nn.init.constant_(conv1.weight[0][0], 0.25)
nn.init.constant_(conv1.weight[0][1], 0.5)
nn.init.constant_(conv1.weight[1][0], 1) 
nn.init.constant_(conv1.weight[1][1], 2) 
out1 = conv1(input) # compute the convolution

conv2 = nn.Conv2d(conv1_out_channels, conv2_out_channels, kernel_size, bias=False)
nn.init.constant_(conv2.weight[0][0], 0.25)
nn.init.constant_(conv2.weight[0][1], 0.5)
nn.init.constant_(conv2.weight[1][0], 1) 
nn.init.constant_(conv2.weight[1][1], 2) 
out2 = conv2(out1) # compute the convolution

for tensor, name in zip([input, conv1.weight, out1, conv2.weight, out2], ['input', 'conv1', 'out1', 'conv2', 'out2']):
    print('{}: {}'.format(name, tensor))
    print('{} shape: {}'.format(name, tensor.shape))

Running this gives the following output:

input: tensor([[[[ 0.,  1.,  2.],
          [ 3.,  4.,  5.],
          [ 6.,  7.,  8.]],

         [[ 9., 10., 11.],
          [12., 13., 14.],
          [15., 16., 17.]]]])
input shape: torch.Size([1, 2, 3, 3])
conv1: Parameter containing:
tensor([[[[0.2500, 0.2500],
          [0.2500, 0.2500]],

         [[0.5000, 0.5000],
          [0.5000, 0.5000]]],


        [[[1.0000, 1.0000],
          [1.0000, 1.0000]],

         [[2.0000, 2.0000],
          [2.0000, 2.0000]]]], requires_grad=True)
conv1 shape: torch.Size([2, 2, 2, 2])
out1: tensor([[[[ 24.,  27.],
          [ 33.,  36.]],

         [[ 96., 108.],
          [132., 144.]]]], grad_fn=<MkldnnConvolutionBackward>)
out1 shape: torch.Size([1, 2, 2, 2])
conv2: Parameter containing:
tensor([[[[0.2500, 0.2500],
          [0.2500, 0.2500]],

         [[0.5000, 0.5000],
          [0.5000, 0.5000]]],


        [[[1.0000, 1.0000],
          [1.0000, 1.0000]],

         [[2.0000, 2.0000],
          [2.0000, 2.0000]]]], requires_grad=True)
conv2 shape: torch.Size([2, 2, 2, 2])
out2: tensor([[[[ 270.]],

         [[1080.]]]], grad_fn=<MkldnnConvolutionBackward>)
out2 shape: torch.Size([1, 2, 1, 1])

Notice how the each channel of the convolution sums over all previous channels outputs.

Answer 8

Refer to "Local Connectivity" section in here and slide 7-18.

"Receptive Field" hyperparameter of filter is defined by height & width only, as depth is fixed by preceding layer's depth.

NOTE that "The extent of the connectivity along the depth axis is always equal to the DEPTH of the input volume" -or- DEPTH of activation map (in case of later layers).

Intuitively, this must be due to the fact that image channels data are interleaved, not planar. This way, applying filter can be achieved simply by column vectors multiplication.

NOTE that Convolutional Network learns all the filter parameters (including the depth dimension) and they are total "hwinput_layer_depth + 1 (bias)".

Answer 9

My understanding from this paper https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/TASLP2339736-proof.pdf was that the filters are used on each input channels (i.e input feature map in the paper) separately and the result is summed, as described in eq. 8. Here they use different filters for each channel, but you could totally use the same filter. As to knowing the software implementation in a particular ML library such as Tensorflow or PyTorch, this requires inspection as suggested by @Mohsin Bukhari

Answer 10

Other answers claim that you have a different 2-dimensional kernel (i.e. a matrix) for each channel. This is not wrong, but it's just a conceptual interpretation of the 2d convolution, which emphasizes that different channels may provide different information during the convolution: this is the only advantage that I see of this interpretation!

Another interpretation (which I think has more advantages, as I will explain below) is to think of a kernel as a 3d multi-dimensional array that has the same depth as the input. So, for example, if you have an input (an image or feature map) of shape $H \times W \times D$, then a single kernel needs to have the shape $K \times L \times D$, where $D$ is the depth of both the input and kernel. If you have more than one kernel, they will all have the depth $D$. This interpretation is consistent with the excerpt that you are quoting.

This second interpretation is conceptually useful when you need to deal with 3d convolutions: if you follow this interpretation, a 3d convolution is just a convolution where kernels don't necessarily have the same depth as the input (and that would be the only difference between 2d and 3d convolutions). The other advantage of this interpretation is that your confusion would not have arisen in the first place: your confusion arouse because you thought that kernels are 2d arrays. Another advantage is that, if you have $N > 1$ kernels (which is the usual case), then you know immediately that $N$ is the depth of the output volume and you don't need to think about multiple channels of the input volume: you can simply think in terms of multi-dimensional arrays.

So, to clarify, in case it wasn't already super clear: no, you don't apply the same 2d matrix to multiple channels of the input!

Answer 11

There are only restriction in 2D. Why?

Imagine a fully connected layer.

It'd be awfully huge, each neuron would be connected to maybe 1000x1000x3 inputs neurons. But we know that processing nearby pixel makes sense, therefore we limit ourselves to a small 2D-neighborhood, so each neuron is connected to only a 3x3 near neurons in 2D. We don't know such a thing about channels, so we connect to all channels.

Still, there would be too many weights. But because of the translation invariance, a filter working well in one area is most probably useful in a different area. So we use the same set of weights across 2D. Again, there's no such translation invariance between channels, so there's no such restriction there.