0
$\begingroup$

After looking into transformer-based models that used multiple stacked encoders and decoders, I am trying to understand how cross attention in the decoders work. In a transformer with a single encoder/decoder, my understanding is the queries come from the decoder, while the keys and values come from the output of the encoder.

But when multiple stacked encoders are used, do the decoders only attend to the output of the final encoder layer? Or do they attend to all the encoder layers?

To put it visually, does cross attention look like this?

Attends hidden state of final encoder layer

Or like this?

Attends all encoder layers

$\endgroup$

1 Answer 1

0
$\begingroup$

The decoder uses the output of only the final encoder layer for cross attention. The first diagram is correct.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .