13 votes
Accepted

Why are the initial weights of neural networks randomly initialised?

You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue. http://www.deeplearningbook.org/contents/optimization.html Perhaps the only property known with complete ...
user avatar
  • 1,390
8 votes
Accepted

Do we know what the units of neural networks will do before we train them?

In reverse order to how you asked: all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal This actually happens if you ...
user avatar
  • 23.8k
7 votes

Why are the initial weights of neural networks randomly initialised?

The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to ...
user avatar
5 votes
Accepted

How are the kernels initialized in a convolutional neural network?

The kernels are usually initialized at a seemingly arbitrary value, and then you would use a gradient descent optimizer to optimize the values, so that the kernels solve your problem. There are many ...
user avatar
5 votes
Accepted

Is random initialization of the weights the only choice to break the symmetry?

Randomising just b sort of works, but setting w to all zero causes severe problems with vanishing gradients, especially at the ...
user avatar
  • 23.8k
5 votes
Accepted

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

There seem to be two different ideas in this question here: What's the impact / importance of our choice for reward values? What's the impact / importance of our choice for initial value estimates (...
user avatar
  • 9,379
4 votes
Accepted

How to solve the problem of too big activations when using genetic algorithms to train neural networks?

Your inputs should stay in a low range. Ideally for neural networks, the inputs are normalised to mean 0, standard deviation 1. I suspect this applies equally well to GA-driven NNs as gradient-driven ...
user avatar
  • 23.8k
3 votes

How are newer weight initialization techniques better than zero or random initialization?

There are several ways to answer this question. First of all, there are several mathematical arguments on why using some kind of initialization is better. Consider reading, for example, Xavier et al.. ...
user avatar
  • 381
2 votes

Is random initialization of the weights the only choice to break the symmetry?

Most of the explanations given for choosing something or not choosing something (like hyperparameter tuning) in deep learning are based on empirical studies, like analysing the error over a number of ...
user avatar
  • 532
2 votes

Why are the initial weights of neural networks randomly initialised?

That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, Gradient Descent Finds Global Minima of ...
user avatar
2 votes
Accepted

Why would the "improvement" be the result of random initialization, and so why should we use multiple runs?

Neural networks use random number generators in multiple places. Most notably for weight initialisation, but also for features such as dropout, selecting minibatches within epochs, and train/cv split ...
user avatar
  • 23.8k
2 votes

Why are weights not initialized with mean=1?

Interesting question, I can come with 2 explanations why we don't initialize weights with 1 mean value : It may be easier for the network to learn identity function, but we may have a similar issue ...
user avatar
  • 211
2 votes
Accepted

How should I initialize the weights of the neural network so that the initial policy is uniform?

You may be interested in section 3.2 of this paper What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020) by Google Research. They claim that the initialization of the ...
user avatar
  • 876
2 votes
Accepted

Weights initialization once the Neural Network is trained

Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of ...
user avatar
1 vote

What is the analytical formula for "Kaiming He" probability density function?

The activation function proposed by He et al. is not a new probability function of its own kind. It's an improvement over a previously proposed activation function now called Xavier or Glorot (even ...
user avatar
1 vote
Accepted

How to explain that a same DNN model have radically different behaviours with each new initialization and training?

I'm sure the biases are initially initialized to zero but I don't know how the weights are handled. Looking at the Dense layer docs: by default Dense layers biases are initialized with zeros (...
user avatar
  • 1,813
1 vote

Are there any downsides of using a fixed seed for a neural network's weight initialization?

When you use a particular seed, it actually ceases to become a random initialization and is instead fixed. I believe the only reason to actually do this would be for reliable reproduction in research ...
user avatar
  • 139

Only top scored, non community-wiki answers of a minimum length are eligible