How are the kernels initialized in a convolutional neural network?

Question

I am currently learning about CNNs. I am confused about how filters (aka kernels) are initialized.

Suppose that we have a $3 \times 3$ kernel. How are the values of this filter initialized before training? Do you just use predefined image kernels? Or are they randomly initialized, then changed with backpropagation?

Answer 1

The kernels are usually initialized at a seemingly arbitrary value, and then you would use a gradient descent optimizer to optimize the values, so that the kernels solve your problem.

There are many different initialization strategies.

• Set all values to a constant (for example, zero)
• Sample from a distribution, such as a normal or uniform distribution
• There are also some heuristic methods that seem to work very well in practice; a popular one is the so-called Glorot initializer, which is named after Xavier Glorot, who introduced them here. Glorot initializers also sample from distribution, but they truncate the values based on the kernel complexity.
• For specific types of kernels, there are other defaults that seem to perform well. See for example this paper.

Exploring initialization strategies is something I do when my model is not able to converge (gradient problems) or when the training seems to be stuck for a long time before the loss function starts to decrease. These are signs that there might be a better initialization strategy to look for.