Questions tagged [weights-initialization]

For questions about the different techniques of initializing weights (or parameters) of machine learning models.

Filter by
Sorted by
Tagged with
1
vote
1answer
27 views

Why are weights not initialized with mean=1?

I wonder why weights are initialized with zero-mean. It is one of the reasons, why deep architectures cannot be trained without skip connections. Without the skip connections, the zero initialization ...
1
vote
0answers
18 views

Which methods for weight initialization in Neural Networks are currently common practice?

I am currently researching the topic of weight initialization methods for (deep) neural networks and I'm a little stuck. The result of my work should be an overview of methods that are currently in ...
0
votes
1answer
39 views

How to explain that a same DNN model have radically different behaviours with each new initialization and training?

I'm trying to predict the continuous values of a variable $y$ using a Fully Connected Neural Network while providing it with data from a $(3300, 13)$ matrix $X$ where $X[i, :]=[0,...,1,...,0,x_{i}]$. ...
0
votes
0answers
38 views

Why would my neural network have either an accuracy of 90% or 10% on the validation data, given a random initialization?

I'm making a custom neural network framework (in C++, if that is of any help). When I train the model on MNIST, depending on how happy the network is feeling, it'll give me either 90%+ accuracy, or ...
4
votes
0answers
47 views

Can the quality of randomness in neural network initialization affect model fitting?

This is a topic I have been arguing about for some time now with my colleagues, maybe you could also voice your opinion about it. Artificial neural networks use random weight initialization within a ...
1
vote
1answer
38 views

Bounding Box Regression - An Adventure in Failure

I've solved many problems with neural networks, but rarely work with images. I have about 18 hours into creating a bounding box regression network and it continues to utterly fail. With some loss ...
2
votes
0answers
47 views

Are there any new weight initialization techniques for DNN published after 2015?

Considering weights initialization in my personal projects, I always used some standard techniques such as: Glorot (also known as Xavier) initialization (2010). Mertens initialization (2010). He ...
3
votes
0answers
55 views

Why is there a Uniform and Normal version of He / Xavier initialization in DL libraries?

Two of the most popular initialization schemes for neural network weights today are Xavier and He. Both methods propose random weight initialization with a variance dependent on the number of input ...
1
vote
0answers
38 views

Why should variance(output) equal variance(input) in Xavier Initialisation?

In a lot of explanations online for Xavier Initialization, I see the following: With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
0
votes
1answer
27 views

Are there any downsides of using a fixed seed for a neural network's weight initialization?

For example, if we set the random seed to be 0, will we run into any problems? For example, maybe for seed 0, we can only reach a certain training error, but other seeds will converge to a much lower ...
1
vote
0answers
152 views

What is the justification for Kaiming He initialization?

I've been trying to understand where the formulas for Xavier and Kaiming He initialization come from. My understanding is that these initialization schemes come from a desire to keep the gradients ...
2
votes
0answers
41 views

How efficient is SCAWI weight initialization method?

I'm currently in the middle of a project (for my thesis) constructing a deep neural network. Since I'm still in the research part, I'm trying to find various ways and techniques to initialize weights. ...
3
votes
1answer
53 views

How are newer weight initialization techniques better than zero or random initialization?

How do newer weight initialization techniques (He, Xavier, etc) improve results over zero or random initialization of weights in a neural network? Is there any mathematical evidence behind this?
2
votes
1answer
161 views

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

Is there any empirical/theoretical evidence on the effect of initial values of state-action and state values on the training of an RL agent (the values an RL agent assigns to visited states) via MC ...
4
votes
1answer
1k views

Do we know what the units of neural networks will do before we train them?

I was learning about back-propagation and, looking at the algorithm, there is no particular 'partiality' given to any unit. What I mean by partiality there is that you have no particular ...
6
votes
1answer
132 views

How to solve the problem of too big activations when using genetic algorithms to train neural networks?

I am trying to create a fixed-topology MLP from scratch (with C#), which can solve some simple problems, such as the XOR problem and MNIST classification. The network will be trained purely with ...
13
votes
3answers
2k views

Why are the initial weights of neural networks randomly initialised?

This might sound silly to someone who has plenty of experience with neural networks but it bothers me... Random initial weights might give you better results that would be somewhat closer to what a ...