Skip to main content

New answers tagged

1 vote
Accepted

Can transformer attention make predictions based on analogy?

To complete the pattern based on the given examples, the last statement should follow the same structure. Each example follows a clear pattern where the second person in the first part of the sentence ...
Kulin Patel's user avatar
0 votes

Can transformer attention make predictions based on analogy?

After some testing, I believe the answer to the question is no. From training the following sentences set: Asdfogiug likes Zsdfoiusdhf and Zsdfoiusdhf likes [Asdfogiug too]. Bsodifhas likes ...
James's user avatar
  • 157
0 votes

probability intepreter of attention mechanism in Seq2Seq

Pretty much anything can be interpreted as a distribution or some kind of energy model, that doesn't mean that everything is the same First of all, the softmax output of a Seq2Seq is just a ...
Alberto's user avatar
  • 2,273
1 vote
Accepted

Why are the Q and K matrices two separate matrices in attention?

Your calculations are correct, though training a whole $W$ is not always equivalent. We can agree that $W_q, W_k \in R^{n\times m}$, thus $W_{qk} = W_qW_k^T \in R^{n\times n}$ Now, check the parameter ...
Alberto's user avatar
  • 2,273
0 votes

Why don't people use nonlinear activation functions after projecting the query key value in attention?

here are a couple of reasons why nonlinear activation functions aren't typically used after projecting the query, key, and value vectors in attention mechanisms like those found in transformers: ...
Emerson Sanchotene's user avatar

Top 50 recent answers are included