As far as I can tell, BERT is a type of Transformer architecture. What I do not understand is:

  1. How is Bert different from the original transformer architecture?

  2. What tasks are better suited for BERT, and what tasks are better suited for the original architecture?


1 Answer 1


What is a transformer?

The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-called attention (i.e. a mechanism that determines the importance of words to other words in a sentence or which words are more likely to come together) and the non-use of recurrent connections (or recurrent neural networks) to solve tasks that involve sequences (or sentences), even though RNN-based systems were becoming the standard practice to solve natural language processing (NLP) or understanding (NLU) tasks. Hence the name of the paper "Attention is all you need", i.e. you only need attention and you don't need recurrent connections to solve NLP tasks.

Both the encoder-decoder architecture and the attention mechanism are not novel proposals. In fact, previous neural network architectures to solve many NLP tasks, such as machine translation, had already used these mechanisms (for example, take a look at this paper). The novelty of the transformer and this cited paper is that it shows that we can simply use attention to solve tasks that involve sequences (such as machine translation) and we do not need recurrent connections, which is an advantage, given that recurrent connections can hinder the parallelization of the training process.

The original transformed architecture is depicted in figure 1 of the cited paper. Both the encoder and decoder are composed of

  • attention modules
  • feed-forward (or fully connected) layers
  • residual (or skip) connections
  • normalization layers
  • dropout
  • label smoothing
  • embedding layers
  • positional encoding

The decoder part is also composed of a linear layer followed by a softmax to solve the specific NLP task (for example, predict the next word in a sentence).

What is BERT?

BERT stands for Bidirectional Encoder Representations from Transformers, so, as the name suggests, it is a way of learning representations of a language that uses a transformer, specifically, the encoder part of the transformer.

What is the difference between the transformer and BERT?

  • BERT is a language model, i.e. it represents the statistical relationships of the words in a language, i.e. which words are more likely to come after another word and stuff like that. Hence the part Representations in its name, Bidirectional Encoder Representations from Transformers.

    BERT can be trained in an unsupervised way for representation learning, and then we can fine-tune BERT on the so-called downstream tasks in a supervised fashion (i.e. transfer learning). There are pre-trained versions of BERT that can be already fine-tuned (e.g. this one) and used to solve your specific supervised learning task. You can play with this TensorFlow tutorial to use a pre-trained BERT model.

    On the other hand, the original transformed was not originally conceived to be a language model, but to solve sequence transduction tasks (i.e. converting one sequence to another, such as machine translation) without recurrent connections (or convolutions) but only attention.

  • BERT is only an encoder, while the original transformer is composed of an encoder and decoder. Given that BERT uses an encoder that is very similar to the original encoder of the transformer, we can say that BERT is a transformer-based model. So, BERT does not use recurrent connections, but only attention and feed-forward layers. There are other transformed-based neural networks that use only the decoder part of the transformer, for example, the GPT model.

  • BERT uses different hyper-parameters than the ones used in Attention is all you need to achieve the best performance. For example, it uses 12 and 16 "attention heads" (please, read the transformer paper to know more about these "attention heads") rather than 8 (although in the original transformer paper the authors experimented with a different number of heads).

  • BERT also uses segment embeddings, while the original transformer only uses word embeddings and positional encodings.

There are probably other small differences that I missed, but, after having read the paper Attention is all you need and quickly read some parts of the BERT paper, these seem to be the main differences.

When to use BERT and the transformer?

Although I never used them, I would say that you want to use BERT whenever you want to solve an NLP task in a supervised fashion, but your labeled training dataset is not big enough to achieve good performance. In that case, you start with a pre-trained BERT model, then fine-tune it with your small labeled dataset. You probably need to add specific layers to BERT to solve your task.

  • 2
    $\begingroup$ Well explained! $\endgroup$
    – avocado
    Jul 25, 2021 at 22:00
  • 1
    $\begingroup$ You said it was supposed to solve seq2seq tasks without convolution, but it does use convolution doesn't it? Right after the attention mechanism dense layers are convolved across the attention outputs + positional embeddings, albeit not with the traditional conventions of CNNs (e.g. there is no flexible stride). $\endgroup$
    – profPlum
    Mar 13, 2022 at 23:57
  • 1
    $\begingroup$ @profPlum There are no convolutions in the transformer. The original paper even says "Since our model contains no recurrence and no convolution...". $\endgroup$
    – nbro
    Mar 14, 2022 at 8:54
  • 2
    $\begingroup$ @nbro There are convolutions, though the authors call them "Point-wise feed-forward networks" and they discard the traditional conventions of CNNs (hence your quote). BUT the original paper also said this: "Another way of describing this [point-wise feedforward network] is as two convolutions with kernel size 1." $\endgroup$
    – profPlum
    Mar 18, 2022 at 23:35
  • 2
    $\begingroup$ @profPlum Thanks for sharing this info. As you can imagine, I gave this answer already a long time ago, so I forgot the details of the paper. However, note that you could also view a "normal" fully connected layer as a convolution layer with a kernel that has the same dimensions as the input. See also this. $\endgroup$
    – nbro
    Mar 19, 2022 at 0:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .