I'm currently in the middle of a project (for my thesis) constructing a deep neural network. Since I'm still in the research part, I'm trying to find various ways and techniques to initialize weights. Obviously, every way will be evaluated and we will choose the one that fits best with our data set and our desired outcome.
I'm all familiar with the Xavier initialization, the classic random one, the He initialization, and zeros. Searching through papers I came across the SCAWI one (Statistically Controlled Activation Weight Initialization). If you have used this approach, how efficient is it?
(Also, do you know any good sources to find more of these?)