Timeline for What's make transformer encoder difference from its decoder part?
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 1 at 8:53 | vote | accept | Muhammad Ikhwan Perwira | ||
| Dec 1 at 8:44 | comment | added | cinch | While the decoder processes sequences in parallel during training by using teacher forcing where the target sequence is fed as input, this still involves extra complexity due to masking compared to the fully parallel encoder. And during inference, decoder is significantly less efficient compared to the encoder since the token-by-token generation cannot be parallelized which is a bottleneck for tasks like text generation or machine translation. Hope now clarifies. | |
| Dec 1 at 8:40 | comment | added | Muhammad Ikhwan Perwira | So, you said decoder is not efficient in computation compared to encoder? Is it during inference or training? | |
| Dec 1 at 8:36 | comment | added | cinch | If you apply causal mask correctly in encoder then it's enough, but you can also leverage decoder's inherent causal autoregressive feature for MLM, at least in principle without considering parallelism efficiency. | |
| Dec 1 at 8:34 | comment | added | Muhammad Ikhwan Perwira | For MLM, isn't encoder only enough? | |
| Dec 1 at 7:44 | comment | added | cinch | Not necessarily, though most decoder transformers such as GPT/Mistral/LLaMA are designed specifically for token-by-token causal generation, they sometimes can also be used in non-generative contexts if causal constraints are appropriate for your task, such as Masked Language Modeling (MLM) or some ML interpretability problems involving understanding how earlier steps influence later ones. For classification problems or NER, certainly you don't use causal decoder-only transformer but encoder to have better bidirectional contextualization and more efficient parallelism. Hope this clarifies. | |
| Dec 1 at 7:27 | comment | added | Muhammad Ikhwan Perwira | So, the transformer decoder is strictly be used for autoregressive generation only? When my problem didn't have to include generation, I shouldn't use decoder? | |
| Dec 1 at 6:56 | history | edited | cinch | CC BY-SA 4.0 |
added 28 characters in body
|
| Nov 30 at 20:03 | history | answered | cinch | CC BY-SA 4.0 |