Why are the initial weights of neural networks randomly initialised?

Question

This might sound silly to someone who has plenty of experience with neural networks but it bothers me...

Random initial weights might give you better results that would be somewhat closer to what a trained neural network should look like, but it might as well be the exact opposite of what it should be, while 0.5, or some other average for the range of reasonable weights' values, would sound like a good default setting.

Why are the initial weights of neural networks randomly initialized rather than being all set to, for example, 0.5?

Answer 1

You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue.

http://www.deeplearningbook.org/contents/optimization.html

Perhaps the only property known with complete certainty is that the initial parameters need to “break symmetry” between different units. If two hidden units with the same activation function are connected to the same inputs, then these units must have different initial parameters. If they have the same initial parameters, then a deterministic learning algorithm applied to a deterministic cost and model will constantly update both of these units in the same way. Even if the model or training algorithm is capable of using stochasticity to compute different updates for different units (for example, if one trains with dropout), it is usually best to initialize each unit to compute a different function from all of the other units. This may help to make sure that no input patterns are lost in the null space of forward propagation and no gradient patterns are lost in the null space of back-propagation.

Answer 2

The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to the same value. While not all of the methods to train neural networks are gradient based, most of them are, and it has been shown in several cases that initializing the neural network to the same value makes the network take much longer to converge on an optimum solution. Also, if you want to retrain your neural network because it got stuck in a local minima, it will get stuck in the same local minima. For the above reasons, we do not set the initial weights to a constant value.

References: Why doesn't backpropagation work when you initialize the weights the same value?

Answer 3

That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, Gradient Descent Finds Global Minima of Deep Neural Networks, A Convergence Theory for Deep Learning via Over-Parameterization or Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks). All of the proofs assume that the initial weights are assigned randomly according to a Gaussian distribution. The main reasons this initial distribution is important for the proofs are:

1. Random weights make the ReLU operators in each layer statistically compressive mapping (up to a linear transformation).

2. Random weights preserve separation of input for any input distribution - that is if input samples are distinguishable network propagation will not make them indistinguishable.

Those properties are very difficult to reproduce with deterministically generated initial weight matrices, and even if they are reproducible with deterministic matrices NULL-space (from which we can generate adversarial examples) would likely make the method less useful in practice. More importantly, preservation of those properties during gradient descent would likely make method impractical. But overall it's very difficult but not impossible, and may warrant some research in that direction. In analogous situation, there were some results for Restricted Isometry Property for deterministic matrices in a compressed sensing.

Answer 4

It is okay to initialize the weights to zero for a simple logistic regression, but for a neural network to initialize the weights to parameters to all zero and then apply gradient descent, it won't work.

Assume here that you have in a 2-layered network with just 2 input features and in the hidden layer you have 2 nodes, and the weights are initialized as [u, v] for both the nodes.

So, while optimizing the weights, using gradient descent you would compute:

where α is the learning rate and J is the cost function that you want to minimize by optimizing the weights.

By kind of a proof by induction, you should be able to see that after every single iteration of training, your two hidden units are still computing exactly the same function because both hidden units start off computing the same function, have the same influence on the output unit. So even after multiple iterations, the two hidden units are still symmetric. Hence, there's really no point to having more than one hidden unit because they are all computing the same thing.

In summary, no matter how long you run gradient descent, both the two units compute exactly the same function which is not helpful, because you want the different hidden units to compute different functions. The solution to this is to initialize your parameters randomly.

For larger neural networks, say of three features and maybe a very large number of hidden units, a similar argument still works.

Note: It's okay to initialize the bias term to just zeros. Because so long as weights are initialized randomly, you start off with the different hidden units computing different things.

Answer 5

Another theory relevant I think to this question is the “lottery ticket hypothesis”:

• Jonathan Frankle and Michael Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, 2019, https://arxiv.org/abs/1803.03635

Basically, when you have a neural network with a very high number of neurons with randomly initialized weights in the hidden layers, you have a high chance of a subset of the network being close to a good solution to the problem. Then, by applying gradient descent, you can align the output of the whole network to that of the sub-network and fine tune it.

It would be nice if someone could prove this. If there is a proof I would be interested to see it.