13
votes
Accepted
Why are the initial weights of neural networks randomly initialised?
You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue.
http://www.deeplearningbook.org/contents/optimization.html
Perhaps the only property known with complete ...
- 1,390
8
votes
Accepted
Do we know what the units of neural networks will do before we train them?
In reverse order to how you asked:
all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal
This actually happens if you ...
- 23.8k
7
votes
Why are the initial weights of neural networks randomly initialised?
The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to ...
- 833
5
votes
Accepted
How are the kernels initialized in a convolutional neural network?
The kernels are usually initialized at a seemingly arbitrary value, and then you would use a gradient descent optimizer to optimize the values, so that the kernels solve your problem.
There are many ...
5
votes
Accepted
Is random initialization of the weights the only choice to break the symmetry?
Randomising just b sort of works, but setting w to all zero causes severe problems with vanishing gradients, especially at the ...
- 23.8k
5
votes
Accepted
How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?
There seem to be two different ideas in this question here:
What's the impact / importance of our choice for reward values?
What's the impact / importance of our choice for initial value estimates (...
- 9,379
4
votes
Accepted
How to solve the problem of too big activations when using genetic algorithms to train neural networks?
Your inputs should stay in a low range. Ideally for neural networks, the inputs are normalised to mean 0, standard deviation 1. I suspect this applies equally well to GA-driven NNs as gradient-driven ...
- 23.8k
3
votes
How are newer weight initialization techniques better than zero or random initialization?
There are several ways to answer this question. First of all, there are several mathematical arguments on why using some kind of initialization is better. Consider reading, for example, Xavier et al.. ...
- 381
2
votes
Is random initialization of the weights the only choice to break the symmetry?
Most of the explanations given for choosing something or not choosing something (like hyperparameter tuning) in deep learning are based on empirical studies, like analysing the error over a number of ...
- 532
2
votes
Why are the initial weights of neural networks randomly initialised?
That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, Gradient Descent Finds Global Minima of ...
- 655
2
votes
Accepted
Why would the "improvement" be the result of random initialization, and so why should we use multiple runs?
Neural networks use random number generators in multiple places. Most notably for weight initialisation, but also for features such as dropout, selecting minibatches within epochs, and train/cv split ...
- 23.8k
2
votes
Why are weights not initialized with mean=1?
Interesting question,
I can come with 2 explanations why we don't initialize weights with 1 mean value :
It may be easier for the network to learn identity function, but we may have a similar issue ...
- 211
2
votes
Accepted
How should I initialize the weights of the neural network so that the initial policy is uniform?
You may be interested in section 3.2 of this paper What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020) by Google Research.
They claim that the initialization of the ...
- 876
2
votes
Accepted
Weights initialization once the Neural Network is trained
Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of ...
- 944
1
vote
What is the analytical formula for "Kaiming He" probability density function?
The activation function proposed by He et al. is not a new probability function of its own kind. It's an improvement over a previously proposed activation function now called Xavier or Glorot (even ...
- 3,513
1
vote
Accepted
How to explain that a same DNN model have radically different behaviours with each new initialization and training?
I'm sure the biases are initially initialized to zero but I don't know how the weights are handled.
Looking at the Dense layer docs: by default Dense layers biases are initialized with zeros (...
- 1,813
1
vote
Are there any downsides of using a fixed seed for a neural network's weight initialization?
When you use a particular seed, it actually ceases to become a random initialization and is instead fixed. I believe the only reason to actually do this would be for reliable reproduction in research ...
- 139
Only top scored, non community-wiki answers of a minimum length are eligible