As of May 31, 2023, we have updated our Code of Conduct.

New answers tagged

0 votes

Shape of biases in Transformer's Feedforward Network

The biases are vectors, and their shape should be $b_1\in \mathbb R^{d_{ff}}$ and $b_2\in \mathbb R^{d_\text{model}}$. To verify that let's compute the shapes of the above formula (for simplicity we ...
Luca Anzalone's user avatar
1 vote

How Does The Scaled Dot Product's Dimensions Work Out In Mult-Head Attention?

Say you have 3-dimensional tensors Q & K of shape (B, T, C) (for Batch, Time, and Channels). K is transposed. 'Transposing' a 3-d tensor in this case means ...
Robin van Hoorn's user avatar

Top 50 recent answers are included