I wonder why weights are initialized with zero-mean. It is one of the reasons, why deep architectures cannot be trained without skip connections. Without the skip connections, the zero initialization becomes problematic, because the identity function cannot be learned in earlier layers (this is a simplified explanation I know). But why can we not initialize weights around one? This would enhance the intrinsic learning of the identity function. Of course, the skip connections also allow a better backpropagation of the gradients, but couldn't this be helpful anyways? Can anyone tell me, why this is not done?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Interesting question,
I can come with 2 explanations why we don't initialize weights with 1 mean value :
- It may be easier for the network to learn identity function, but we may have a similar issue about not being able to learn comparison, comparison is quite an important reasoning in my opinion, this is why having negative weight values is important, and initializing all weights around 1 may make it difficult for the network to get negative weights.
- Maybe it causes an issue with the non-linearity of the activation functions. If we use ReLU function for example, it only is non linear if there are negative values in the network. It is not uncommon to have positive inputs for a network, so initializing all weights around 1 may lead to have only positives values on the network, which makes ReLU function linear.
Anyway weight initialization has always been a complicated topic and it seems there is no definitive answer to what should be done about initializing weights. I am not an expert of the topic, just giving you my thoughts about it.
