New answers tagged

1 vote

Is there a proper initialization technique for the weight matrices in multi-head attention?

IMO xavier/glorot is the correct way to initialize the $W_Q$ and $W_K$ matrices. In section 3.2.1 of the transformer paper the ...
  • 11
1 vote

Why are the initial weights of neural networks randomly initialised?

Another theory relevant I think to this question is the “lottery ticket hypothesis”: Jonathan Frankle and Michael Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, ...
  • 141

Top 50 recent answers are included