12
You shouldn't assign all to 0.5 because you'd have the "break symmetry" issue.
http://www.deeplearningbook.org/contents/optimization.html
Perhaps the only property known with complete certainty is that the initial
parameters need to “break symmetry” between different units. If two hidden
units with the same activation function are connected to the ...
8
In reverse order to how you asked:
all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal
This actually happens if you initialise the weights equally (e.g. all zero). Gradients in that case are the same to each neuron in the same layer, and everything changes in lockstep. A neural ...
7
The initial weights in a neural network are initialized randomly because the gradient based methods commonly used to train neural networks do not work well when all of the weights are initialized to the same value. While not all of the methods to train neural networks are gradient based, most of them are, and it has been shown in several cases that ...
5
The kernels are usually initialized at a seemingly arbitrary value, and then you would use a gradient descent optimizer to optimize the values, so that the kernels solve your problem.
There are many different initialization strategies.
Set all values to a constant (for example, zero)
Sample from a distribution, such as a normal or uniform distribution
There ...
5
Randomising just b sort of works, but setting w to all zero causes severe problems with vanishing gradients, especially at the start of learning.
Using backpropagation, the gradient at the outputs of a layer L involves a sum multiplying the gradient of the inputs to layer L+1 by the weights (and not the biases) between the layers. This will be zero if the ...
4
There seem to be two different ideas in this question here:
What's the impact / importance of our choice for reward values?
What's the impact / importance of our choice for initial value estimates (how do we initialise our table of $Q(s, a)$ values in the case of a simple, tabular RL algorithm like Sarsa or $Q$-learning)?
The reward values are typically ...
4
Your inputs should stay in a low range. Ideally for neural networks, the inputs are normalised to mean 0, standard deviation 1. I suspect this applies equally well to GA-driven NNs as gradient-driven ones.
Your weights should be both positive and negative.
In addition, once trained, they tend to follow a certain size distribution. It helps if you start ...
3
There are several ways to answer this question. First of all, there are several mathematical arguments on why using some kind of initialization is better. Consider reading, for example, Xavier et al.. Moreover, there are several numerical experiments showing the importance of initialization.
The motivation for Xavier initialization in Neural Networks is to ...
2
Most of the explanations given for choosing something or not choosing something (like hyperparameter tuning) in deep learning are based on empirical studies, like analysing the error over a number of iterations. So, this answer is what people in deep learning side give.
Since you have asked for a mathematical explanation, I suggest you read the paper ...
2
That is a very deep question. There was series of papers recently proving the convergence of gradient descent for overparameterized deep networks (for example, Gradient Descent Finds Global Minima of Deep Neural Networks, A Convergence Theory for Deep Learning via Over-Parameterization or Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU ...
2
Neural networks use random number generators in multiple places. Most notably for weight initialisation, but also for features such as dropout, selecting minibatches within epochs, and train/cv split for cross-validation.
That means that any result metric from the neural network e.g. accuracy, loss, F1 score, is a random variable.
Reporting a single value of ...
2
Interesting question,
I can come with 2 explanations why we don't initialize weights with 1 mean value :
It may be easier for the network to learn identity function, but we may have a similar issue about not being able to learn comparison, comparison is quite an important reasoning in my opinion, this is why having negative weight values is important, and ...
2
You may be interested in section 3.2 of this paper What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020) by Google Research.
They claim that the initialization of the policy is very important to performance, sometimes making a huge (66%) improvement, just from the initialization of the policy.
I'm assuming you already know ...
2
Regarding your first code snippet, there is no weight storing or continuation of training between the different CV folds whatsoever: each model is trained anew with the respective training data of each fold and validated on the validation data. Notice that this is exactly the idea behind cross validation - models trained on different folds are completely ...
1
The activation function proposed by He et al. is not a new probability function of its own kind. It's an improvement over a previously proposed activation function now called Xavier or Glorot (even though it was named by the authors normalized activation in the original paper). The Xavier activation is also simply an activation function and not a new kind of ...
1
I'm sure the biases are initially initialized to zero but I don't know how the weights are handled.
Looking at the Dense layer docs: by default Dense layers biases are initialized with zeros (bias_initializer='zeros') and weights are initialized with Glorot uniform (kernel_initializer='glorot_uniform').
... "unusual" element to point here; I've ...
1
When you use a particular seed, it actually ceases to become a random initialization and is instead fixed. I believe the only reason to actually do this would be for reliable reproduction in research and not as a method of training production models.
Only top voted, non community-wiki answers of a minimum length are eligible