New answers tagged weights
1
vote
Is there a proper initialization technique for the weight matrices in multi-head attention?
IMO xavier/glorot is the correct way to initialize the $W_Q$ and $W_K$ matrices.
In section 3.2.1 of the transformer paper the ...
- 11
1
vote
Why are the initial weights of neural networks randomly initialised?
Another theory relevant I think to this question is the “lottery ticket hypothesis”:
Jonathan Frankle and Michael Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, ...
- 141
Top 50 recent answers are included