Questions tagged [weights-initialization]

For questions about the different techniques of initializing weights (or parameters) of machine learning models.

Filter by
Sorted by
Tagged with
2
votes
0answers
38 views

Why is there a Uniform and Normal version of He / Xavier initialization in DL libraries?

Two of the most popular initialization schemes for neural network weights today are Xavier and He. Both methods propose random weight initialization with a variance dependent on the number of input ...
1
vote
0answers
25 views

Why should variance(output) equal variance(input) in Xavier Initialisation?

In a lot of explanations online for Xavier Initialization, I see the following: With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
0
votes
1answer
23 views

Are there any downsides of using a fixed seed for a neural network's weight initialization?

For example, if we set the random seed to be 0, will we run into any problems? For example, maybe for seed 0, we can only reach a certain training error, but other seeds will converge to a much lower ...
2
votes
0answers
63 views

What is the justification for Kaiming He initialization?

I've been trying to understand where the formulas for Xavier and Kaiming He initialization come from. My understanding is that these initialization schemes come from a desire to keep the gradients ...
2
votes
0answers
38 views

How efficient is SCAWI weight initialization method?

I'm currently in the middle of a project (for my thesis) constructing a deep neural network. Since I'm still in the research part, I'm trying to find various ways and techniques to initialize weights. ...
3
votes
1answer
47 views

How are newer weight initialization techniques better than zero or random initialization?

How do newer weight initialization techniques (He, Xavier, etc) improve results over zero or random initialization of weights in a neural network? Is there any mathematical evidence behind this?
2
votes
1answer
103 views

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

Is there any empirical/theoretical evidence on the effect of initial values of state-action and state values on the training of an RL agent (the values an RL agent assigns to visited states) via MC ...
12
votes
3answers
1k views

Why are the initial weights of neural networks randomly initialised?

This might sound silly to someone who has plenty of experience with neural networks but it bothers me... Random initial weights might give you better results that would be somewhat closer to what a ...