4
$\begingroup$

I was recently asked at an interview to calculate the number of parameters for a convolutional layer. I am deeply ashamed to admit I didn't know how to do that, even though I've been working and using CNN for years now.

Given a convolutional layer with ten $3 \times 3$ filters and an input of shape $24 \times 24 \times 3$, what is the total number of parameters of this convolutional layer?

$\endgroup$
4
$\begingroup$

What are the parameters in a convolutional layer?

The (learnable) parameters of a convolutional layer are the elements of the kernels (or filters) and biases (if you decide to have them). There are 1d, 2d and 3d convolutions. The most common are 2d convolutions, which are the ones people usually refer to, so I will mainly focus on this case.

2d convolutions

Example

If the 2d convolutional layer has $10$ filters of $3 \times 3$ shape and the input to the convolutional layer is $24 \times 24 \times 3$, then this actually means that the filters will have shape $3 \times 3 \times 3$, i.e. each filter will have the 3rd dimension that is equal to the 3rd dimension of the input. So, the 3rd dimension of the kernel is not given because it can be determined from the 3rd dimension of the input.

2d convolutions are performed along only 2 axes (x and y), hence the name. Here's a picture of a typical 2d convolutional layer where the depth of the kernel (in orange) is equal to the depth of the input volume (in cyan).

enter image description here

Each kernel can optionally have an associated scalar bias.

At this point, you should already be able to calculate the number of parameters of a standard convolutional layer. In your case, the number of parameters is $10 * (3*3*3) + 10 = 280$.

A TensorFlow proof

The following simple TensorFlow (version 2) program can confirm this.

import tensorflow as tf


def get_model(input_shape, num_classes=10):
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Input(shape=input_shape))
    model.add(tf.keras.layers.Conv2D(10, kernel_size=3, use_bias=True))
    model.add(tf.keras.layers.Flatten())
    model.add(tf.keras.layers.Dense(num_classes))

    model.summary()

    return model


if __name__ == '__main__':
    input_shape = (24, 24, 3)
    get_model(input_shape)

You should try setting use_bias to False to understand how the number of parameters changes.

General case

So, in general, given $M$ filters of shape $K \times K$ and an input of shape $H \times W \times D$, then the number of parameters of the standard 2d convolutional layer, with scalar biases, is $M * (K * K * D) + M$ and, without biases, is $M * (K * K * D)$.

See also these related questions How is the depth of filters of hidden layers determined? and In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?.

1d and 3d convolutions

There are also 1d and 3d convolutions.

For example, in the case of 3d convolutions, the kernels may not have the same dimension as the depth of the input, so the number of parameters is calculated differently for 3d convolutional layers. Here's a diagram of 3d convolutional layer, where the kernel has a depth different than the depth of the input volume.

enter image description here

See e.g. Intuitive understanding of 1D, 2D, and 3D convolutions in convolutional neural networks.

| improve this answer | |
$\endgroup$
  • $\begingroup$ The diagrams were taken from ai.stackexchange.com/a/13786/2444. $\endgroup$ – nbro Mar 17 at 0:36
  • 1
    $\begingroup$ Wow! Simply wow! Thank you very much for the answer. I am normally a visual guy and this is what works, seeing how the data flows. I still have a lot of questions, but this isn't the place as you have more than answered my question. $\endgroup$ – Ælex Mar 17 at 10:34
3
$\begingroup$

For a standard convolution layer, the weight matrix will have a shape of (out_channels, in_channels, kernel_sizes*) in addition you will need a vector of shape [out_channels] for biases. For your specific case, 2d, your weight matrix will have a shape of (out_channels, in_channels, kernel_size[0], kernel_size[1]). Now if we plugin the numbers:

  • out_channels = 10, you're having 10 filters
  • in_channels = 3 the picture is RGB in this case so there are 3 channels (the last dimension of the input)
  • kernel_size[0] = kernel_size[1] = 3

In total you're gonna have 10*3*3*3 + 10 = 280 parameters.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.