2
$\begingroup$

The BERT paper has the following diagram (Figure 3):

enter image description here

It's captioned "Differences in pre-training model architectures". However, I thought the BERT architecture was just a stack of attention blocks like this (from huggingface):

enter image description here

My question: What is the BERT diagram supposed to mean?

My best guess is that it represents the process of predicting an output sequence from a given input sequence. In other words, in the GPT model, one first samples with a sequence length of 1 to get next token T1, then we embed T1 via E2 and sample with a sequence of length 2 to get T2, and so on.

In BERT, I assume we mask all input tokens but the first, then sample the whole thing.

The problem is that I don't understand what the Trm blocks mean, or how this relates to the BERT architecture.

$\endgroup$

1 Answer 1

1
$\begingroup$
  1. "BertBase" is basically what's inside the ellipses labelled "Trm" (attention heads + FF) - both for BERT and GPT.

  2. The main difference is that BERT pays bidirectional attention while GPT is unidirectional (from left to right) or "causal".

  3. T_N of GPT is the contextualized last token (embedding) which is used to calculate the next token.

$\endgroup$
1
  • $\begingroup$ So in general each Trm block outputs something of shape (Batch, Seqlen, d), except in the output where each T_i selects the final sequence element? $\endgroup$ Feb 20 at 14:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .