What are the techniques used to initialize weights for neural networks?

Question

When creating a neural network to predict the impact of risks on the project cost, what techniques are used to initialize the weights provided to the hidden layers and the output layer?

Answer 1

Both Keras and PyTorch provide a range of initialization classes and functions. Of these, probably the two most commonly used are the Glorot (Xavier in Pytorch) and He (Kaiming in pytorch) initializers. Additionally, biases are commonly initialized to 0 - see for example this answer on Stack Overflow: Initial bias values for a neural network.

Glorot initialization (Glorot and Bengio: Understanding the difficulty of training deep feedforward neural networks Xavier) was designed for NNs that use sigmoid or tanh activation functions.

He initialization (He at al.: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification) was designed for NNs that use rectifiers such as RELU.

Each of these has a normal and uniform variant - i.e. the initial weights are chosen by sampling from a normal or uniform distribution with a mean of zero. They vary in the way the standard deviation of the normal distribution or limits of the uniform distribution are calculated. The Glorot initializer is based on the sum of the input and output units to the weight tensor, whereas the He initializer just uses the number of input units.

Answer 2

Both Keras and Torch provide a wide range of techniques for weight initialization. Like Zero initialization :- Initializing all weights to zero it is easy way to initialize the weights and biases but it is not very effective since all neuron learns same things and predicts same things which leads to symmetry breaking issue.

One initialization :- This method is as same as zero initialization in this method instead of zeros we initialize all weights with zeros but again it also leads to symmetry breaking down.

Random Initialization :- In this method weights are initialize based on normal or uniform distribution. This method is common and mostly use in initialization of the weights of neurons. and it also solves the problems like stucking at local minima. but there are chances of gradient vanishing and gradient exploding.

As per Lynn said you can use the initialization like He and glorot which are used for specific types of layers which gives much better performance of that specific types of layers.

For more information you can check this links

for Keras :https://keras.io/api/layers/initializers/

for pytorch:https://pytorch.org/docs/stable/nn.init.html

Answer 3

Yes, weight initialization in the network has had a high impact on the model. The choice of weight initialization depends on a few factors like the architecture of the neural network, activation function, and input data. you can share those things that help us help you with which activation function to use. e.g. if you are using Deep Network with ReLu then He initialization is a good choice, for a network with Sigmoid and Tanh use XavierGlorot.

Here I am listing a few Weight initialization techniques and when to use them:

Zero:

• All Weights are initialized to Zero.
• Mostly it is not recommended because it stops effective learning, because all the layers are going to learn almost the same features while training.

Random:

• Weights are initialized with random values with the distribution of normal or uniform.
• It is better to break symmetry but it leads to vanishing and exploding gradient. Mostly in POCs, we start from this weighting technique.

Xavier/Glorot:

• Weights are in distribution with 0 mean and std. deviation of the sqrt(2/(inputs+outputs))
• Useful when networks use tanh or sigmoid activation because it helps in maintaining variance of the activation function and back prop gradient

He:

• It is similar to the Xavier with 0 mean and std. deviation of the sqrt(2/inputs).
• Ideals for networks like ReLU and variants. It helps in mitigation of the vanishing gradient.

Orthogonal:

• In this weights are orthogonal metrics. This helps in norm preservation.
• Best suited for RNN because it helps in maintaining gradient magrnitude in deeper networks.

Sparse:

• Some of the weights are set to non-zero values while others are zero.
• Useful in case of sparse network data type handing.