I am a bit confused on what cross attention mechanisms are doing. I understand that the currently decoded output is usually the query and the conditioning/input (from an encoder) is the key and value. The query is multiplied by the key to make an attention matrix that details how much each element of the currently decoded output relates to each element of the input and then this matrix is applied to the value. Isn't the output of this cross attention mechanism an enriched version of the input and not the currently decoded output since the attention matrix is being applied to values. I would think that the output of the entire mechanism should be and enriched version of the currently decoded output based on what the input is.

A side thought of mine is that the cross attention mechanism does in fact produce an enriched version of the input which is then added back into the currently decoded output through a skip connection.



You must log in to answer this question.

Browse other questions tagged .