I am currently learning about CNN's and I am confused on how filter/kernels are initialized beside their size? Say if you want a filter of 3x3 how are the inner values initialized a the start?


Do you just use those predefined image-kernels as a start? Or are they randomly initialized and retrained from backpropagation? I am pretty confused on this matter because so far of all the lecture I have taken no one really talk about this yet and I want to know it now.

  • $\begingroup$ Welcome to AI! I took the liberty of editing out your addendum. Feel free to re-edit the question, or, self-answer as you find gain new information. $\endgroup$ – DukeZhou Jan 22 '18 at 22:37

The kernels are usually initialized at a seemingly arbitrary value and then you would use a gradient descent optimizer to optimize the values so that the kernels solve your problem.

There are many different initialization strategies.

  • Set all values to 1 or 0 or another constant
  • Sample from a distribution, such as a normal or uniform distribution
  • There are also some heuristic methods that seem to work very well in practice, a popular one is the so-called glorot initializer named after Xavier Glorot who introduced them here. Glorot initializers also sample from a distribution but truncate the values based on the kernel complexity.
  • For specific types of kernels, there are other defaults that seem to perform well. See for example this paper.

Exploring initialization strategies is something I do when my model is not able to converge (gradient problems) or when the training seems to be stuck for a long time before the loss function starts to decrease. These are signs that there might be a better initialization strategy to look for.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.