1
$\begingroup$

Transformers use just matrices to transform input embeddings, which is halfway to being a connected dense layer (add a bias and activation). So, why don't transformers have dense layers for encoding input into Query Key Value?

$\endgroup$

1 Answer 1

0
$\begingroup$

I suppose the main reason is that they were proposed as a more parallelizable alternative to RNNs that is quicker to compute. Hence, making them just big matrix multiplications achieves this and the lack of intermediate activations is just replaced by the sheer number of transformers that focus their attention on different parts of the input sequence.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .