Confusion about conversion of RGB image to grayscale image using a convolutional layer with 2-dimensional filters

Let us imagine $$x$$ as a tensor containing 1000 RGB images, each of size $$64 \times 32$$.

>>> x = torch.randn(1000, 3, 64, 32)
>>> print(x.shape)
torch.Size([1000, 3, 64, 32])


I am using a 2d convolutional layer that converts RGB images to single channel (say grayscale) images

>>> in_ch = 3
>>> out_ch = 1
>>> m = nn.Conv2d(in_ch, out_ch, 3, 1, 1)
>>> print(m)
Conv2d(3, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))


I passed the tensor $$x$$ in the convolutional layer and obtained another tensor of 1000 grayscale images, each of size $$64 \times 32$$.

>>> output = m(x)
>>> print(output.shape)
torch.Size([1000, 1, 64, 32])


Now, I can say that my convolutional layer converted an RGB image into a grayscale image using 2d kernel.

How it is doing?

RGB image has 3 planes each of size $$64 \times 32$$. If a kernel of 2 dimensions is used, then we will get 3 planes in output, corresponding to R, G, and B. How is it possible to convert an image with 3 channels into an image with one channel using 2d kernel?

I can visualize easily if I use a 3d kernel since the kernel considers three channels simultaneously and produces a single feature map for an RGB image.

• Aug 2 at 22:32

Apparently we don't count the number of channels as a dimension, because we mostly care about pixels and not channels.

Note the groups parameter of Conv2d, which affects how the channels are convolved. The default is 1, which means:

At groups=1, all inputs are convolved to all outputs.

If you set it to 3 (and 3 output channels) then the Conv2d layer would maintain the channel separation.

Yes, it is really a 3D convolution (in this case), but as I said, we mostly care about pixel dimensions. You could think of each pixel as being made up of 3 numbers, rather than a separate dimension.

• Okay! seems that kernel size is misleading. Actually the kernel size is not $3 \times 3$. It is $3 \times 3 \times 3$. Aug 2 at 22:32
• (+1) for Yes, it is really a 3D convolution (in this case), because I need to understand remaining to appreciate the answer further. Aug 2 at 23:30