New answers tagged weights
1
vote
Is there a proper initialization technique for the weight matrices in multi-head attention?
IMO xavier/glorot is the correct way to initialize the $W_Q$ and $W_K$ matrices.
In section 3.2.1 of the transformer paper the ...
- 11
1
vote
Why are the initial weights of neural networks randomly initialised?
Another theory relevant I think to this question is the “lottery ticket hypothesis”:
Jonathan Frankle and Michael Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, ...
- 141
Top 50 recent answers are included
Related Tags
weights × 83neural-networks × 39
machine-learning × 15
deep-learning × 13
convolutional-neural-networks × 13
weights-initialization × 11
training × 9
backpropagation × 9
classification × 5
reference-request × 5
pytorch × 5
reinforcement-learning × 4
performance × 4
artificial-neuron × 4
natural-language-processing × 3
gradient-descent × 3
deep-neural-networks × 3
transformer × 3
attention × 3
filters × 3
logistic-regression × 3
activation-functions × 3
feedforward-neural-networks × 3
tensorflow × 2
terminology × 2