After looking into transformer-based models that used multiple stacked encoders and decoders, I am trying to understand how cross attention in the decoders work. In a transformer with a single encoder/decoder, my understanding is the queries come from the decoder, while the keys and values come from the output of the encoder.
But when multiple stacked encoders are used, do the decoders only attend to the output of the final encoder layer? Or do they attend to all the encoder layers?
To put it visually, does cross attention look like this?
Or like this?
