0
$\begingroup$

Let's consider language translation and let $I_1,\ldots,I_{N_i}$ be the $N_i$ input tokens. My understanding is that the encoder produces $N_i$ embeddings, which I will refer to as $E_1,\ldots,E_{N_i}$. In what follows, let's consider what happens at inference time and not during training. My understanding is that the decoder has multiple inputs and they possibly change when he produces the different output tokens. If considering a single decoder module, I guess it makes sense to distinguish between the external inputs that are fed into the multi-head attention module and the ones that go into the masked multi-head attention module as shown below.
enter image description here
Here are now my questions:

  1. Is it correct that what comes from the left into the multi-head attention modules are always the encoder embeddings $E_1,\ldots,E_{N_i}$?
  2. I believe to understand that the bottom input to the masked multi-head attention module is not always the same and depends on the iteration we find ourselves in, i.e. which output token we predict and which decoder module is considered if there are multiple. More precisely, is it true that for the first prediction, i.e. when predicting the first output token, the sole input to the first decoder module is the embedding of the 'START' token plus positional encoding?
  3. What are the bottom inputs of the different decoder modules when predicting the individual output tokens?
Improve this question
$\endgroup$
1
  • $\begingroup$ Please, ask only one question per post, even if they are related. As a rule of thumb, if you can't put a single question in the title that summaries your problem/question, then it means you need to split the post into multiple ones. "Understanding the transformer at inference time" looks like a too big problem for a single post. So, maybe you could change your title to ask such specific question. Thanks. $\endgroup$
    – nbro
    Commented Nov 17, 2023 at 0:44

1 Answer 1

Reset to default
1
$\begingroup$

1 - yes, they are considered the conditioning of the model, so they are always known

2&3 - at inference, you start with a single <SOS> token, which will be forwarded through the decoder, and conditioned on the encoder output, to get a final softmax distribution at the end; at this point, you sample a token from such distribution, you concatenate it to the input to get <SOS>sampled_token, and you feed this new string to the decoder (which will be than forwarded, conditioned, and will produce a softmax from which you'll sample the next) and the end is just determined when the sampled token is the end of sentece <EOS>

Improve this answer
$\endgroup$
2
  • $\begingroup$ Wouldn't the decoder produce two word predictions in the 2nd iteration, three in the 3rd and so on then? Would you then just take the one for the new word? So ignore the first prediction in the 2nd iteration, ignore the first two in the 3rd and so on? $\endgroup$
    – Felix Crazzolara
    Commented Nov 14, 2023 at 14:28
  • $\begingroup$ @FelixCrazzolara depends on how you treat the input, but no, if you just input a string, you predict only the next, it's during training that you have that behavior because you want to train it in one forward pass $\endgroup$
    – Alberto
    Commented Nov 14, 2023 at 17:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .