Considering weights initialization in my personal projects, I always used some standard techniques such as:

  1. Glorot (also known as Xavier) initialization (2010).
  2. Mertens initialization (2010).
  3. He initialization (2015).

As it is a very active research field, are there some innovations in recent years that have increased the performance of DNNs?

I am thinking specifically of architectures such as DNNs and CNNs with activation functions, such as ReLU, ELU, PReLU, Leaky ReLU, SELU, Swish, and Mish.

  • 1
    $\begingroup$ I found this paper Improving Weight Initialization of ReLU and Output Layers (2019). There's only 1 citation for now (so I don't know how much it's reliable), but, according to their abstract, they obtained good results. You can find the source code here. Maybe you could have a look at it, and later provide an answer to your own question (if nobody does it). $\endgroup$ – nbro Jan 20 at 21:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.