New answers tagged linear-algebra
0
votes
Accepted
Shape of biases in Transformer's Feedforward Network
The biases are vectors, and their shape should be $b_1\in \mathbb R^{d_{ff}}$ and $b_2\in \mathbb R^{d_\text{model}}$.
To verify that let's compute the shapes of the above formula (for simplicity we ...
- 1,213
1
vote
How Does The Scaled Dot Product's Dimensions Work Out In Mult-Head Attention?
Say you have 3-dimensional tensors Q & K of shape (B, T, C) (for Batch, Time, and Channels).
K is transposed. 'Transposing' a 3-d tensor in this case means ...
- 1,574
Top 50 recent answers are included