1
$\begingroup$

Encoders are commonly used for understanding sequences (like classification, NER, etc.), while decoders are used for generative tasks in an autoregressive manner (e.g., GPT-style text generation).

Decoders are autoregressive because they generate one token at a time, conditioning on previously generated tokens.

In contrast, encoders process the entire input sequence in a single pass and return contextualized representations of the tokens.

If I train the encoder part of a Transformer using a sliding window approach, where the model uses the output of the previous inference as input for the next step (similar to a feedback loop), wouldn't it behave in a way that's similar to a decoder's autoregressive behavior?

In this setup, I'm essentially giving the model the last predicted output as the next input, so wouldn't this iterative process have similar properties to autoregressive generation in a decoder?

For example Xt, yt here to train "transformer encoder only model" for every timestep:

x1: [I, <NEXT>, <PAD>, <PAD>, <PAD>]
y1: [I, LOVE, <NEXT>, <PAD>, <PAD>]

x2: [I, LOVE, <NEXT>, <PAD>, <PAD>]
y2: [I, LOVE, KANADE, <NEXT>, <PAD>]

x3: [I, LOVE, KANADE, <NEXT>, <PAD>]
y3: [I, LOVE, KANADE, <EOT>, <PAD>]

x4: [I, LOVE, KANADE, <EOT>, <PAD>]
y4: [I, LOVE, KANADE, <EOT>, <PAD>]

x5 and so on: same as x4
y5 and so on: same as x4

Both encoders and decoders use self-attention mechanisms. Given that self-attention is responsible for capturing dependencies between tokens, what are the key bounds or differences between the two?

Is the primary difference just that the decoder generates tokens one by one, while the encoder processes the whole sequence at once?

I find this situation reminiscent of the traditional autoencoder structure, where the encoder maps the input to a latent space, and the decoder maps this latent space back into a human-readable representation. But, transformer encoder-decoder model is unintuitive for me.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You’re right that encoder-decoder transformer aligns with the traditional autoencoder (AE) structure except AE’s encoder output is usually a compressed latent representation while transformer’s encoder output is an enriched contextualized representation.

While your sliding window approach makes an encoder behave similarly to a decoder, it lacks causal constraints in the sense that your encoder processes input tokens in parallel, not sequentially. This can introduce dependencies that violate autoregressive constraints, for instance, in your above window 2 the encode can attend to <NEXT> token to predict the next token. Also transformer decoders are optimized for token-by-token autoregressive generation, while your sliding windows require reprocessing overlapping inputs which can be computationally expensive.

Improve this answer
$\endgroup$
6
  • $\begingroup$ So, the transformer decoder is strictly be used for autoregressive generation only? When my problem didn't have to include generation, I shouldn't use decoder? $\endgroup$
    – Muhammad Ikhwan Perwira
    Commented Dec 1 at 7:27
  • $\begingroup$ Not necessarily, though most decoder transformers such as GPT/Mistral/LLaMA are designed specifically for token-by-token causal generation, they sometimes can also be used in non-generative contexts if causal constraints are appropriate for your task, such as Masked Language Modeling (MLM) or some ML interpretability problems involving understanding how earlier steps influence later ones. For classification problems or NER, certainly you don't use causal decoder-only transformer but encoder to have better bidirectional contextualization and more efficient parallelism. Hope this clarifies. $\endgroup$
    – cinch
    Commented Dec 1 at 7:44
  • $\begingroup$ For MLM, isn't encoder only enough? $\endgroup$
    – Muhammad Ikhwan Perwira
    Commented Dec 1 at 8:34
  • $\begingroup$ If you apply causal mask correctly in encoder then it's enough, but you can also leverage decoder's inherent causal autoregressive feature for MLM, at least in principle without considering parallelism efficiency. $\endgroup$
    – cinch
    Commented Dec 1 at 8:36
  • $\begingroup$ So, you said decoder is not efficient in computation compared to encoder? Is it during inference or training? $\endgroup$
    – Muhammad Ikhwan Perwira
    Commented Dec 1 at 8:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .