After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can only look at prior tokens.

Why does GPT-2 not require the encoder part of the original transformer architecture?

GPT-2 architecture with only decoder layers

enter image description here

  • $\begingroup$ You say "GPT-2 essentially uses only the encoder part of the original transformer architecture", but maybe you meant "decoder"? $\endgroup$ – nbro Mar 27 at 20:42
  • 1
    $\begingroup$ @nbro Correct, it shoud be "decoder", sorry for the mistake, updated the question. $\endgroup$ – Athena Wisdom Mar 27 at 20:53

GPT-2 is a close copy of the basic transformer architecture.

GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the MASKING in the multi-head attention block, the decoder is only allowed to glean information from the prior words in the sentence. It works just like a traditional language model as it takes word vectors as input and produces estimates for the probability of the next word as outputs but it is auto-regressive as each token in the sentence has the context of the previous words. Thus GPT-2 works one token at a time.

BERT, by contrast, is not auto-regressive. It uses the entire surrounding context all-at-once. GPT-2 the context vector is zero-initialized for the first word embedding.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.