Considering weights initialization in my personal projects, I always used some standard techniques such as:
- Glorot (also known as Xavier) initialization (2010).
- Mertens initialization (2010).
- He initialization (2015).
As it is a very active research field, are there some innovations in recent years that have increased the performance of DNNs?
I am thinking specifically of architectures such as DNNs and CNNs with activation functions, such as ReLU, ELU, PReLU, Leaky ReLU, SELU, Swish, and Mish.