Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [weights-initialization]

For questions about the different techniques of initializing weights (or parameters) of machine learning models.

Filter by
Sorted by
Tagged with
0 votes
1 answer
44 views

Xavier vs He initialization with tanh

I'm a student and in the lecture, I learned that He initialization is better than Xavier if you use ReLU activation function. In addition, I also learned that Xavier initialization is better than He ...
COTHE's user avatar
  • 5
2 votes
1 answer
67 views

Is it a requirement/recommendation to normalize my inputs into [0,1] range?

All features of my input dataset, which is going to be used for training a simple multi-layered neural network, are in range $[-1,+1]$ and the output of $NN$ is a single number again in range $[-1,+1]$...
Bikay's user avatar
  • 23
2 votes
2 answers
126 views

How Xavier Initialization formula works

In the research paper on Xavier initialization what is the purpose of putting $n_{in}$ and $n_{out}$ under 2 and added them it just says as a compromise but it is not exactly a harmonic mean or an ...
Stef's user avatar
  • 95
0 votes
1 answer
74 views

Why CNN filters (kernels) are randomly initialized?

I learned that when CNN filters are defined, they are initialized with random weights and bias(Im not sure about bias). Then as learning step goes on, the weight values change and each filter makes ...
COTHE's user avatar
  • 5
4 votes
1 answer
148 views

If we had to choose between Uniform(0,1) and Uniform(-1,0), which one would you expect to work best and why?

I'm working with a fully connected neural network with input 32x32x3. The architecture includes a dense layer 32 + ReLu activation, then another dense layer 64 + ReLu Activation, followed by a ...
Miguel's user avatar
  • 43
0 votes
3 answers
135 views

What are the techniques used to initialize weights for neural networks?

When creating a neural network to predict the impact of risks on the project cost, what techniques are used to initialize the weights provided to the hidden layers and the output layer?
maya sy's user avatar
  • 11
2 votes
1 answer
62 views

How can I deal with random weights initialisation when predicting a time-series sine function?

I am training a simple RNN model in keras to predict a time series. The time series I am considering is just a sine function The task to solve is the following: ...
apt45's user avatar
  • 123
2 votes
0 answers
69 views

Is orthogonal initialization still useful when hidden layer sizes vary?

Pytorch's orthogonal initialization cites "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks ", Saxe, A. et al. (2013), which gives as reason for the ...
Gabi's user avatar
  • 121
1 vote
2 answers
324 views

What is the impact of the initialization of weights in the performance of a neural network in machine learning?

In my own experience, weight initialization matters for model convergence. Theoretically, can different weight initialization methods eventually converge to the same optimal solution? Are their ...
Robin van Hoorn's user avatar
3 votes
1 answer
666 views

Initial State of RNN

Can I initialize the initial state of my RNN to be non-zero? I have some initial condition of the sequence and I want to use this initial condition as the initial state.
wrek's user avatar
  • 183
1 vote
0 answers
47 views

Distributions over outputs for randomly initialized neural networks

Does anyone have any pointers to resources about the properties of randomly initialized neural networks (with no training)? I'm guessing this might depend on the network architecture and ...
mdc's user avatar
  • 380
1 vote
1 answer
47 views

When do you know that your neural network is learning something when metrics are garbage?

When training a neural network for binary classification on a highly imbalanced set its training loss decreases, however validation loss increase even though accuracy is very high (due to highly ...
haneulkim's user avatar
  • 113
0 votes
1 answer
22 views

Why do smaller weights converge faster for RNNs?

I am writing a Recurrent Neural Network using only the NumPy library for a binary classification problem. When I initialize the weights with np.random.randn, after 1000 epochs it gets ~60% accuracy, ...
user avatar
1 vote
1 answer
194 views

Neural network: Initial weights for layer with non-negative constraint

I wonder how to initialize the weights of a layer with non-negative weight constraints and sigmoid activation afterwards. I did not find any guidelines. I thought about taking inspiration of ...
Philipp123's user avatar
2 votes
1 answer
78 views

How to decode P bits that represent a random weight generator?

So I've been tasked by my neural network professor at university to replicate the following research: Intelligent Breast Cancer Diagnosis Using Hybrid GA-ANN. Each chromosome represents a possible net,...
JOSEPH CAROÈ's user avatar
0 votes
1 answer
1k views

How should I initialize the weights of the neural network so that the initial policy is uniform?

I would like to train a neural network (NN) so that it learns the policy and value function for my agent. Since I am using reinforcement learning and do not want to prefer certain actions in certain ...
Druudik's user avatar
  • 149
1 vote
1 answer
569 views

Weights initialization once the Neural Network is trained

I am trying to understand how weights are initialized in a Neural Network using Keras deep learning framework and what happens if I train a Neural Network and then I want to train it again: are the ...
HelpNeederStudent's user avatar
1 vote
0 answers
25 views

Why Acme is using own uniform initializer?

Why is Acme using own initializer for both tanh and ELU, when commonly used for tanh is Xavier and for ELU is He initializer? What mathematics is behind them? Here is the code. ...
Bc. Martin Kubovčík's user avatar
1 vote
0 answers
18 views

Are there any recommendations on initialising a single parameter in deep learning?

I want to initialize a parameter, which is a single real number in my model. If you want the role of the parameter in the model, you can assume it as the parameter to multiply with the output of the ...
hanugm's user avatar
  • 3,880
0 votes
0 answers
23 views

In general, when are the normal, uniform and zero initializers used?

I came across a Conv2D layer in a fully convolutional network, which used a kernel_initializer='zero' for regression. Why is a ...
skinnybb's user avatar
1 vote
1 answer
227 views

What is the analytical formula for "Kaiming He" probability density function?

A probability density function is a real-valued function that roughly gives the density of probability at a particular value of a random variable. For example, the probability density function of a ...
hanugm's user avatar
  • 3,880
1 vote
0 answers
78 views

What are strategies for data driven weights initialization?

I am beginner in deep learning and currently training a few neural networks (Pytorch) for problems in audio and speech. For my tasks, simple feed-forward networks are working well enough. I use basic ...
Axeon Thra's user avatar
8 votes
1 answer
4k views

Is there a proper initialization technique for the weight matrices in multi-head attention?

Self-attention layers have 4 learnable tensors (in the vanilla formulation): Query matrix $W_Q$ Key matrix $W_K$ Value matrix $W_V$ Output matrix $W_O$ Nice illustration from https://jalammar....
spiridon_the_sun_rotator's user avatar
0 votes
1 answer
98 views

Why would the "improvement" be the result of random initialization, and so why should we use multiple runs?

I got this feedback for my thesis paper. The improvement shown in the results section could be the result of random initialization. There should be multiple runs with means and standard deviations. ...
Md. Asif Iqbal Fahim's user avatar
0 votes
0 answers
71 views

Why don't we use this intialization with SGD rather than random?

Suppose I have a loss function as a polynomial with its variables being the weights of a network I wish to tune. Now, we want to find the minima of the loss function - so basically ...
neel g's user avatar
  • 146
1 vote
1 answer
61 views

Why are weights not initialized with mean=1?

I wonder why weights are initialized with zero-mean. It is one of the reasons, why deep architectures cannot be trained without skip connections. Without the skip connections, the zero initialization ...
spadel's user avatar
  • 31
1 vote
0 answers
31 views

Which methods for weight initialization in Neural Networks are currently common practice?

I am currently researching the topic of weight initialization methods for (deep) neural networks and I'm a little stuck. The result of my work should be an overview of methods that are currently in ...
Josi M.'s user avatar
  • 11
0 votes
1 answer
63 views

How to explain that a same DNN model have radically different behaviours with each new initialization and training?

I'm trying to predict the continuous values of a variable $y$ using a Fully Connected Neural Network while providing it with data from a $(3300, 13)$ matrix $X$ where $X[i, :]=[0,...,1,...,0,x_{i}]$. ...
Daviiid's user avatar
  • 575
0 votes
0 answers
215 views

Why would my neural network have either an accuracy of 90% or 10% on the validation data, given a random initialization?

I'm making a custom neural network framework (in C++, if that is of any help). When I train the model on MNIST, depending on how happy the network is feeling, it'll give me either 90%+ accuracy, or ...
Ilknur Mustafa's user avatar
4 votes
0 answers
65 views

Can the quality of randomness in neural network initialization affect model fitting?

This is a topic I have been arguing about for some time now with my colleagues, maybe you could also voice your opinion about it. Artificial neural networks use random weight initialization within a ...
Aki Koivu's user avatar
0 votes
1 answer
299 views

Bounding Box Regression - An Adventure in Failure [closed]

I've solved many problems with neural networks, but rarely work with images. I have about 18 hours into creating a bounding box regression network and it continues to utterly fail. With some loss ...
David Hoelzer's user avatar
2 votes
0 answers
143 views

Are there any new weight initialization techniques for DNN published after 2015?

Considering weights initialization in my personal projects, I always used some standard techniques such as: Glorot (also known as Xavier) initialization (2010). Mertens initialization (2010). He ...
Stefano Barone's user avatar
8 votes
1 answer
373 views

Why is there a Uniform and Normal version of He / Xavier initialization in DL libraries?

Two of the most popular initialization schemes for neural network weights today are Xavier and He. Both methods propose random weight initialization with a variance dependent on the number of input ...
Tinu's user avatar
  • 628
2 votes
0 answers
302 views

Why should variance(output) equal variance(input) in Xavier Initialisation?

In a lot of explanations online for Xavier Initialization, I see the following: With each passing layer, we want the variance to remain the same. This helps us keep the signal from exploding to a ...
THAT_AI_GUY's user avatar
0 votes
1 answer
892 views

Are there any downsides of using a fixed seed for a neural network's weight initialization?

For example, if we set the random seed to be 0, will we run into any problems? For example, maybe for seed 0, we can only reach a certain training error, but other seeds will converge to a much lower ...
user3180's user avatar
  • 618
5 votes
0 answers
833 views

What is the justification for Kaiming He initialization?

I've been trying to understand where the formulas for Xavier and Kaiming He initialization come from. My understanding is that these initialization schemes come from a desire to keep the gradients ...
Jack M's user avatar
  • 262
3 votes
0 answers
72 views

How efficient is SCAWI weight initialization method?

I'm currently in the middle of a project (for my thesis) constructing a deep neural network. Since I'm still in the research part, I'm trying to find various ways and techniques to initialize weights. ...
ChrisP's user avatar
  • 41
3 votes
1 answer
132 views

How are newer weight initialization techniques better than zero or random initialization?

How do newer weight initialization techniques (He, Xavier, etc) improve results over zero or random initialization of weights in a neural network? Is there any mathematical evidence behind this?
jaeger6's user avatar
  • 308
3 votes
1 answer
2k views

How does the initialization of the value function and definition of the reward function affect the performance of the RL agent?

Is there any empirical/theoretical evidence on the effect of initial values of state-action and state values on the training of an RL agent (the values an RL agent assigns to visited states) via MC ...
user avatar
4 votes
1 answer
1k views

Do we know what the units of neural networks will do before we train them?

I was learning about back-propagation and, looking at the algorithm, there is no particular 'partiality' given to any unit. What I mean by partiality there is that you have no particular ...
Htnamus's user avatar
  • 43
6 votes
3 answers
1k views

Is random initialization of the weights the only choice to break the symmetry?

My knowledge Suppose you have a layer that is fully connected, and that each neuron performs an operation like a = g(w^T * x + b) were ...
gvgramazio's user avatar
7 votes
1 answer
378 views

How to solve the problem of too big activations when using genetic algorithms to train neural networks?

I am trying to create a fixed-topology MLP from scratch (with C#), which can solve some simple problems, such as the XOR problem and MNIST classification. The network will be trained purely with ...
Joshua Jang's user avatar
6 votes
1 answer
5k views

How are the kernels initialized in a convolutional neural network?

I am currently learning about CNNs. I am confused about how filters (aka kernels) are initialized. Suppose that we have a $3 \times 3$ kernel. How are the values of this filter initialized before ...
Inkplay_'s user avatar
  • 421
16 votes
5 answers
3k views

Why are the initial weights of neural networks randomly initialised?

This might sound silly to someone who has plenty of experience with neural networks but it bothers me... Random initial weights might give you better results that would be somewhat closer to what a ...
Matas Vaitkevicius's user avatar