Skip to main content
Share Your Experience: Take the 2024 Developer Survey
13 votes
Accepted

Why are the initial weights of neural networks randomly initialised?

You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue. http://www.deeplearningbook.org/contents/optimization.html Perhaps the only property known with complete ...
SmallChess's user avatar
  • 1,411
8 votes
Accepted

Do we know what the units of neural networks will do before we train them?

In reverse order to how you asked: all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal This actually happens if you ...
Neil Slater's user avatar
  • 32.5k
7 votes

Why are the initial weights of neural networks randomly initialised?

The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to ...
Aiden Grossman's user avatar
5 votes
Accepted

If we had to choose between Uniform(0,1) and Uniform(-1,0), which one would you expect to work best and why?

It is not the input to first layer you need to worry about, but the output from the hidden layer to the next layer. No matter how the inputs and weights are arranged, after passing through ReLU in the ...
Neil Slater's user avatar
  • 32.5k
5 votes
Accepted

How are the kernels initialized in a convolutional neural network?

The kernels are usually initialized at a seemingly arbitrary value, and then you would use a gradient descent optimizer to optimize the values, so that the kernels solve your problem. There are many ...
Mustafa Radha's user avatar
5 votes
Accepted

Is random initialization of the weights the only choice to break the symmetry?

Randomising just b sort of works, but setting w to all zero causes severe problems with vanishing gradients, especially at the ...
Neil Slater's user avatar
  • 32.5k
5 votes

Why are the initial weights of neural networks randomly initialised?

That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, Gradient Descent Finds Global Minima of ...
mirror2image's user avatar
5 votes
Accepted

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

There seem to be two different ideas in this question here: What's the impact / importance of our choice for reward values? What's the impact / importance of our choice for initial value estimates (...
Dennis Soemers's user avatar
  • 10.3k
4 votes

Is there a proper initialization technique for the weight matrices in multi-head attention?

IMO xavier/glorot is the correct way to initialize the $W_Q$ and $W_K$ matrices. In section 3.2.1 of the transformer paper the ...
pi-tau's user avatar
  • 805
4 votes
Accepted

How to solve the problem of too big activations when using genetic algorithms to train neural networks?

Your inputs should stay in a low range. Ideally for neural networks, the inputs are normalised to mean 0, standard deviation 1. I suspect this applies equally well to GA-driven NNs as gradient-driven ...
Neil Slater's user avatar
  • 32.5k
3 votes

How are newer weight initialization techniques better than zero or random initialization?

There are several ways to answer this question. First of all, there are several mathematical arguments on why using some kind of initialization is better. Consider reading, for example, Xavier et al.. ...
hola's user avatar
  • 381
3 votes
Accepted

What is the impact of the initialization of weights in the performance of a neural network in machine learning?

Progress about how to best initialize the weights, is what has made neural networks to be popular again. Initially (around the 80s I think), NNs were initialized from Normal distributions like $\...
Luca Anzalone's user avatar
3 votes
Accepted

How should I initialize the weights of the neural network so that the initial policy is uniform?

You may be interested in section 3.2 of this paper What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020) by Google Research. They claim that the initialization of the ...
Taw's user avatar
  • 1,261
3 votes
Accepted

Weights initialization once the Neural Network is trained

Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of ...
desertnaut's user avatar
  • 1,044
3 votes
Accepted

Why would the "improvement" be the result of random initialization, and so why should we use multiple runs?

Neural networks use random number generators in multiple places. Most notably for weight initialisation, but also for features such as dropout, selecting minibatches within epochs, and train/cv split ...
Neil Slater's user avatar
  • 32.5k
3 votes
Accepted

Is it a requirement/recommendation to normalize my inputs into [0,1] range?

Generally, between -1 and 1 are ideal, though you can get away with a wider range. For example, using the z-score as the range, you will be outside of this range, sometimes by quite a bit (say, -30 ...
David Hoelzer's user avatar
2 votes

Why are the initial weights of neural networks randomly initialised?

It is okay to initialize the weights to zero for a simple logistic regression, but for a neural network to initialize the weights to parameters to all zero and then apply gradient descent, it won't ...
ksgr5566's user avatar
  • 121
2 votes

Is random initialization of the weights the only choice to break the symmetry?

Most of the explanations given for choosing something or not choosing something (like hyperparameter tuning) in deep learning are based on empirical studies, like analysing the error over a number of ...
varsh's user avatar
  • 562
2 votes

Why are weights not initialized with mean=1?

Interesting question, I can come with 2 explanations why we don't initialize weights with 1 mean value : It may be easier for the network to learn identity function, but we may have a similar issue ...
Ubikuity's user avatar
  • 211
2 votes

Initial State of RNN

Can I initialize the initial state of my RNN to be non-zero? Yes, the initial state / weights of a neural network can be initialized to non-zero values. In fact, the ...
Snehal Patel's user avatar
2 votes
Accepted

How Xavier Initialization formula works

More of an extended comment but it felt worth giving as an answer. Equations (10) and (11) give you the estimates $\mathrm{Var}(W^i) = \frac{1}{n_i}$ and $\mathrm{Var}(W^i) = \frac{1}{n_{i+1}}$ ...
Paul VanKoughnett's user avatar
1 vote

What is the impact of the initialization of weights in the performance of a neural network in machine learning?

Weight initialization can and often does matter, hence why pre-training language models are useful for downstream tasks. A randomly initialized model is not guaranteed to converge, especially if the ...
Thomas K's user avatar
1 vote

What are the techniques used to initialize weights for neural networks?

Both Keras and Torch provide a wide range of techniques for weight initialization. Like Zero initialization :- Initializing all weights to zero it is easy way to initialize the weights and biases but ...
Keval's user avatar
  • 111
1 vote

What are the techniques used to initialize weights for neural networks?

Both Keras and PyTorch provide a range of initialization classes and functions. Of these, probably the two most commonly used are the Glorot (Xavier in Pytorch) and He (Kaiming in pytorch) ...
Lynn's user avatar
  • 141
1 vote

How can I deal with random weights initialisation when predicting a time-series sine function?

From comments, you are testing your ideas with a very small neural networks. The highly variable end result with large dependency on initial conditions is a common result of working with small numbers ...
Neil Slater's user avatar
  • 32.5k
1 vote
Accepted

When do you know that your neural network is learning something when metrics are garbage?

You ask different questions in the title and in the body of the question, so I'll start from the title. When do you know your neural network is learning? The surprisingly simply answer to this ...
Edoardo Guerriero's user avatar
1 vote
Accepted

Why do smaller weights converge faster for RNNs?

There is no magic value that work for every network but in general: too large initial weights lead to exploding gradients (i.e. no convergence) too small initial weights lead to vanishing gradients (...
Edoardo Guerriero's user avatar
1 vote

Neural network: Initial weights for layer with non-negative constraint

I, unfortunately, cannot provide you with a scientifically based answer, so I'll try to logic my way to an answer. I know that there are things called 'nonnegativity-constrained autoencoders'. I do ...
Robin van Hoorn's user avatar
1 vote

What is the analytical formula for "Kaiming He" probability density function?

The activation function proposed by He et al. is not a new probability function of its own kind. It's an improvement over a previously proposed activation function now called Xavier or Glorot (even ...
Edoardo Guerriero's user avatar
1 vote
Accepted

How to explain that a same DNN model have radically different behaviours with each new initialization and training?

I'm sure the biases are initially initialized to zero but I don't know how the weights are handled. Looking at the Dense layer docs: by default Dense layers biases are initialized with zeros (...
Kostya's user avatar
  • 2,534

Only top scored, non community-wiki answers of a minimum length are eligible