1
$\begingroup$

What is the difference between an encoder-decoder transformer and decoder-only transformer with regard to the loss calculation. Specifically, how does the loss signal differ? And how does this relate to token efficiency?

As far as I understand, an encoder-decoder model is composed of an encoder and a decoder. The output of the first is fed as queries and keys into a second multi-head attention layer of the decoder part. The input to the encoder is the input sequence. The input to the decoder is the target sequence, shifted right by one, and the output is a probability distribution over the tokens.

A decoder-only model is basically the same minus the multi-head attention layer that takes the encoder as input. Hence, as far as I understand, the inputs and outputs to the decoder-only model are identical to the encoder-decoder model's decoder.

When training a decoder-only model, I am given a sequence of, say, 4097 tokens. The first 4096 tokens are my input sequence, the last 4096 tokens my target labels. That way, the model will relate each token to its successor in the sequence. So, when training, will I calculate the loss signal of 4096 tokens at the same time? Shouldn't this be identical to the encoder-decoder model?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

For autoregressive tasks like language modeling, decoder-only models can process long sequences in a straightforward way and avoid the encoder step entirely. The CE loss is calculated for each token prediction across the sequence except the first one in parallel, 4096 predictions from your 4097 token inputs example, and then summed or averaged across all positions to get the total loss to back propagate.

In encoder-decoder models, the CE loss is calculated for each token in the target sequence conditioned on the input from the encoder, which requires cross-attention. Each token in the target sequence learns to predict the next one based on both the input sequence and preceding target tokens. Therefore encoder-decoder models are less token-efficient for training language models, as they split attention across two sequences (input and target). They are better suited for translation-like tasks where there's an explicit mapping between input and output sequences required.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks very much! One thing that's not clear to me is how splitting the attention across two sequences leads to lower token efficiency. Do you have more details on this? $\endgroup$
    – Green grün 绿色 vert зеленый
    Commented Nov 14 at 6:38
  • $\begingroup$ @Greengrün绿色vertзеленый During the cross-attention phase, each token in the decoder attends to all the tokens in the encoder’s self-attended output, where decoder might process some tokens in a way that doesn't maximize their utility. For example, the decoder is focusing on tokens in the encoder that may not carry much new information for a given decoding step, thus consuming more resources for potentially less useful information. Of course it's unavoidable for classic encode-decode models. Hope this clarifies. $\endgroup$
    – cinch
    Commented Nov 14 at 7:12
  • $\begingroup$ I see. Thanks for the clarification. $\endgroup$
    – Green grün 绿色 vert зеленый
    Commented Nov 14 at 9:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .