0
$\begingroup$

I have a question regarding the original transformer implementation (as in "Attention is all you need").
Assuming I want to translate English to German.
  1. In the Decoder part, in the self-attention layers, the input is only the German tokens the model already translated, but never the future tokens. So in the input matrices X=K=Q=V, what are the values in those rows? some placeholder null token?
  2. If so, what happens in the (not self) attention afterwards? How does its Q matrix look like as a result from the previous self attention? What bothers me is that its Q matrix would have null/random-valued rows, which would affect the attention result.

Much appreciated.

$\endgroup$
0

2 Answers 2

2
$\begingroup$

In the implementation of transformers, there are specific methods employed to address this issue. The attention mechanism initially establishes a context length, which refers to the number of tokens each attention head processes. The issue you're describing seems to relate to what happens when the number of tokens generated by the decoder is less than the product of the context length and the number of attention heads. This situation is typically resolved by inserting a placeholder token named '[PAD]' for padding purposes. The network, therefore, functions as intended, and the attention calculations include the embeddings of this special token that is added in the vocabulary for that specific purpose. Consequently, the Q matrix will not contain null or random-valued rows, allowing the process to continue smoothly.

As a reference, when programming the tokenizer for the transformer with Hugginface's Transformers library, you can specify whether to use padding in the tokenizer class. This will add the '[PAD]' token to the Lookup table dictionary. This is customizable for different applications; for example, you can choose whether to add the paddings to the right or left side.

$\endgroup$
2
$\begingroup$

In addition to the answer by Cesar Ruiz:

In the Transformer model, padding tokens are used to make all sequences the same length. These padding tokens are not actual data and are assigned a logit of -inf so that they have no effect on the model's computations. When the softmax function is applied, the output probability associated with a logit of -inf will be effectively 0. This ensures that the padding tokens do not influence the model's predictions.

For a more detailed explanation, read this.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .