Why do the values in the cross attentional mechanism within a transformer come from the encoder and not from the decoder?

Question

The transformer architecture contains a cross attention mechanism which is enriching the encoder with information from the decoder. The place where this takes place is visualized in the image below:

The cross attention mechanism within the original transformer architecture is implemented in the following way:

The source for the images is this video. Why are the values in this step coming from the encoder instead of from the decoder? Is this where e.g. the actual language translation happens within a transformer?

Answer 1

The transformer architecture contains a cross attention mechanism which is enriching the encoder with information from the decoder. The place where this takes place is visualized in the image below:

I think that you got it the other way round. The encoder passes an enriched input sentence to the decoder. Cross attention helps the decoder to attend to every part of this "enriched" input and produce one output at a time recursively. Initially, the decoder's first prediction is fixed to (start of sentence) token. That gets self attended first, then get attended with encoder's output (the "enriched" input) and gives out a prediction from the word vocab list. This word gets appended to the decoder's input and we repeat the process again.

Answer 2

My intuition is that the decoder essentially performs a dictionary look-up where the encoder acts as the dictionary that provides keys and corresponding values. In this way, the decoder can "ask" for relevant information to fulfill its needs.

Answer 3

The idea of the cross-attention layer is to transform the input words to output words. The Decoder provides context of which input words should we pay attention to next based on the already decoded text.

If the values in the cross-attention came from the decoder as you suggest, we would have to have access to the already translated sentence to translate the sentence, which is absurd.

For example, suppose we are going to translate the sentence "में बैडमिंटन खेलने जा रहा हूँ " in Hindi to English : "I am going to play badminton".

And suppose we are have already translated "I am going to play _" and are looking for the next word : "badminton". In this case the decoder self-attention guesses that the next word is going to be a sports name. The query cross-attention linear layer now transforms the input vector to align with the corresponding vector representation in Hindi for sports names. Now amongst the input words the hindi representation of 'badminton' aligns the most with query vector and thus we know that what the next word to transform to English is.

Now what is left, is to do the actual transformation from Hindi representation of badminton to the English representation. The value linear layer handles that part.

The dot product is able to work because the vector representation of words follows a pattern as shown by the papers referred here : https://kawine.github.io/blog/nlp/2019/06/21/word-analogies.html