How is the number of channels in a convolutional layer shrinked or expanded?

Question

I know in order to shrink or expand the number of channels a 1x1 convolution is performed.

I need to clarify the following: is the 1x1 convolution(s) just a matrix multiplication between the image with shape (hw, 3) (RGB) and a matrix that holds the learnable weights with shape (3, 1)? Which will result in a new matrix of shape (hw, 1) (in this case the number of channels shrunk from 3 to 1).

If the above is correct, what happens under the hood of a NN framework, such as PyTorch, when the number of input channels is equal to the number of output channels? Does a matrix multiplication take place between the input (h*w, 3) and a matrix with learnable weights (num_channels, num_channels)? Doesn't this introduce unnecessary (and unwanted) operations?

Answer 1

It is useless to use a 1x1 convolution if the number of output channel is equal to the number of input channel, as it allows no additionnal expressivity for the network and just adds useless computations. You can however add a ReLu between the two layers to make it useful as the linearity will no longer hold.

If you use a 1x1 convolution (or more precisely, a 1x1xc convolution where c is the number of input channels), then each output channel will be a linear combination of the input channels. The coefficients of these linear combinations corresponds to the coefficients of the different 1x1 kernels.

Answer 2

You can think of applying a point-wise, i.e. $1\times 1$, convolution with $F$ filters to a $H\times W\times C$ image (or feature maps, in general) as the application of a linear fully-connected layer (with $F$ units) to each spatial location of the input, resulting in $H\times W\times F$ tensor.

In general, you can see the application of a convolutional layer (regardless the kernel size) as a matrix multiplication between the kernel matrix, say $K$, and the flattened image matrix, say $I$, where: $K$ is a block diagonal matrix (you have zeros outside the diagonal) build by kind of stacking the kernel (the weights of the conv layer) $H$ times as follows:

then $I$ is obtained by flattening the input image. After the multiplication you reshape the results back to a 3d-tensor shape.

In the example you have a $1\times 2$ kenel (reddish vector), and a $3\times 3$ image. So, you should consider for the matrix multiplication a stack of $2\times 3$ kenels ($k\times W$, in general) and a reshaped image of size $(H\times W,1)$.

what happens under the hood of a NN framework, such as PyTorch, when the number of input channels is equal to the number of output channels?

For practical implementations I don't think the matrix formulation of conv layers is beneficial, especially if you consider GPUs which have many threads and you can basically let each thread access a location of the input tensor, compute the respective index in the kernel, do the product and finally sum. Now if $k=1$ (as in pointwise conv), the matrix $K$ is built from $1\times W$ matrices, stacked $H$ times, resulting in $(W\times H)\times H$ kernel matrix, which is also mostly sparse. Maybe with ad-hoc algorithms for sparse multiplication this operation may be convenient.