How to calculate the number of parameters of a convolutional layer?

Question

I was recently asked at an interview to calculate the number of parameters for a convolutional layer. I am deeply ashamed to admit I didn't know how to do that, even though I've been working and using CNN for years now.

Given a convolutional layer with ten $3 \times 3$ filters and an input of shape $24 \times 24 \times 3$, what is the total number of parameters of this convolutional layer?

Answer 1

What are the parameters in a convolutional layer?

The (learnable) parameters of a convolutional layer are the elements of the kernels (or filters) and biases (if you decide to have them). There are 1d, 2d and 3d convolutions. The most common are 2d convolutions, which are the ones people usually refer to, so I will mainly focus on this case.

2d convolutions

Example

If the 2d convolutional layer has $10$ filters of $3 \times 3$ shape and the input to the convolutional layer is $24 \times 24 \times 3$, then this actually means that the filters will have shape $3 \times 3 \times 3$, i.e. each filter will have the 3rd dimension that is equal to the 3rd dimension of the input. So, the 3rd dimension of the kernel is not given because it can be determined from the 3rd dimension of the input.

2d convolutions are performed along only 2 axes (x and y), hence the name. Here's a picture of a typical 2d convolutional layer where the depth of the kernel (in orange) is equal to the depth of the input volume (in cyan).

Each kernel can optionally have an associated scalar bias.

At this point, you should already be able to calculate the number of parameters of a standard convolutional layer. In your case, the number of parameters is $10 * (3*3*3) + 10 = 280$.

A TensorFlow proof

The following simple TensorFlow (version 2) program can confirm this.

import tensorflow as tf


def get_model(input_shape, num_classes=10):
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Input(shape=input_shape))
    model.add(tf.keras.layers.Conv2D(10, kernel_size=3, use_bias=True))
    model.add(tf.keras.layers.Flatten())
    model.add(tf.keras.layers.Dense(num_classes))

    model.summary()

    return model


if __name__ == '__main__':
    input_shape = (24, 24, 3)
    get_model(input_shape)

You should try setting use_bias to False to understand how the number of parameters changes.

General case

So, in general, given $M$ filters of shape $K \times K$ and an input of shape $H \times W \times D$, then the number of parameters of the standard 2d convolutional layer, with scalar biases, is $M * (K * K * D) + M$ and, without biases, is $M * (K * K * D)$.

See also these related questions How is the depth of filters of hidden layers determined? and In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?.

1d and 3d convolutions

There are also 1d and 3d convolutions.

For example, in the case of 3d convolutions, the kernels may not have the same dimension as the depth of the input, so the number of parameters is calculated differently for 3d convolutional layers. Here's a diagram of 3d convolutional layer, where the kernel has a depth different than the depth of the input volume.

See e.g. Intuitive understanding of 1D, 2D, and 3D convolutions in convolutional neural networks.

Answer 2

For a standard convolution layer, the weight matrix will have a shape of (out_channels, in_channels, kernel_sizes). In addition, you will need a vector of shape [out_channels] for biases. For your specific case, 2d, your weight matrix will have a shape of (out_channels, in_channels, kernel_size[0], kernel_size[1]).

Now, if we plugin the numbers:

• out_channels = 10, you're having 10 filters
• in_channels = 3, the picture is RGB in this case, so there are 3 channels (the last dimension of the input)
• kernel_size[0] = kernel_size[1] = 3

In total you're gonna have 10*3*3*3 + 10 = 280 parameters.