6
$\begingroup$

This is a simple question. I know the weights in a neural network can be initialized in many different ways like: random uniform distribution, normal distribution, and Xavier initialization. But what is the weight initialization trying to achieve?

Is it trying to allow the gradients to be large so it can quickly converge? Is it trying to make sure there is no symmetry in the gradients? Is it trying to make the outputs as random as possible to learn more from the loss function? Is it only trying to prevent exploding and vanishing gradients? Is it more about speed or finding a global maximum? What would the perfect weights (without being learned parameters) for a problem achieve? What makes them perfect? What are the properties in an initialization that makes the network learn faster?

Improve this question
$\endgroup$
5
  • $\begingroup$ Finding the "perfect" anything of a black box model is not feasible. You can use a good initialisation method though. As an example, the goal of Xavier initialisation is to maintain a variance at the end of each layer equal to the incoming variance. This helps dramatically with keeping gradients reasonable $\endgroup$
    – Recessive
    Commented Sep 21, 2020 at 23:24
  • $\begingroup$ @Recessive I’m using perfect very loosely and also I am not asking what the perfect weights would be, but what they would achieve in speeding up learning. $\endgroup$
    – S2673
    Commented Sep 22, 2020 at 0:10
  • $\begingroup$ It also depends on where the line is drawn. Transfer learning is a form of weight initialisation, and in that case that is an exceptional efficient initialiser. If you only consider random initialisation, I would suggest reading this: wandb.com/articles/…. It's quite comprehensive $\endgroup$
    – Recessive
    Commented Sep 22, 2020 at 0:17
  • $\begingroup$ @Recessive I actually just read that, but thanks. I am not only considering random initialization. I am not really looking for a specific strategy, but more what a perfect one would be supposed to achieve. $\endgroup$
    – S2673
    Commented Sep 22, 2020 at 0:21
  • $\begingroup$ This article Why Initialize a Neural Network with Random Weights? should be useful. $\endgroup$
    – nbro
    Commented Sep 22, 2020 at 16:58

2 Answers 2

Reset to default
4
$\begingroup$
  • Is it trying to make sure there is no symmetry in the gradients?

The aim of weight initialization is to make sure that we don't converge to a trivial solution. That's why we have different kinds of initialization depending on the dataset type. So, Yes it is trying to avoid symmetry.

  • Is it trying to allow the gradients to be large so it can quickly converge?

The time it takes to converge, is I think a property of the optimizer and not of the weights initialization. Of course, the manner in which we initialize our weights matters but I think Optimization Algorithms contribute more towards convergence

  • What are the properties in an initialization that makes the network learn faster?

Glorot and Bengio believed that Xavier weight initialization would maintain the variance of activations and back-propagated gradients all the way up or down the layers of a network. Incidentally, when they trained deeper networks that used ReLUs, it was found that a 30-layer CNN using Xavier initialization stalled completely and didn’t learn at all. Thus, it depends on the particular problem at hand.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So the goal is somewhat to get every neuron involved in gradient descent? Making sure none of their gradients are very small or zero? $\endgroup$
    – S2673
    Commented Sep 24, 2020 at 11:20
  • $\begingroup$ Yes, the aim is to have a low number of sparse weights. $\endgroup$
    – Saurav Maheshkar
    Commented Sep 25, 2020 at 13:09
1
$\begingroup$

The most important thing we achieve is indeed making sure the weights are not all equal. If they were, every layer would behave as if it were a single cell.

We typically want weights that are near zero (so unimportant connections will not accidentally dominate) but non-zero.

The different types of initialization all have different motivations, including those mentioned in the question.

If you're curious what the motivation for each one is, I would recommend you check the documentation and try to find the original papers where they were first introduced.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ So I am getting that one motivation is that unimportant connections will not dominate. I will look into the individual initialization strategies, but what would a perfect weight initialization achieve without having contradictory goals? Is it possible to combine the individual initialization goals? $\endgroup$
    – S2673
    Commented Sep 22, 2020 at 0:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .