As far as I can tell, BERT is a type of Transformer architecture. What I do not understand is:
How is Bert different from the original transformer architecture?
What tasks are better suited for BERT, and what tasks are better suited for the original architecture?
The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-called attention (i.e. a mechanism that determines the importance of words to other words in a sentence or which words are more likely to come together) and the non-use of recurrent connections (or recurrent neural networks) to solve tasks that involve sequences (or sentences), even though RNN-based systems were becoming the standard practice to solve natural language processing (NLP) or understanding (NLU) tasks. Hence the name of the paper "Attention is all you need", i.e. you only need attention and you don't need recurrent connections to solve NLP tasks.
Both the encoder-decoder architecture and the attention mechanism are not novel proposals. In fact, previous neural network architectures to solve many NLP tasks, such as machine translation, had already used these mechanisms (for example, take a look at this paper). The novelty of the transformer and this cited paper is that it shows that we can simply use attention to solve tasks that involve sequences (such as machine translation) and we do not need recurrent connections, which is an advantage, given that recurrent connections can hinder the parallelization of the training process.
The original transformer architecture is depicted in figure 1 of the cited paper. Both the encoder and decoder are composed of
The decoder part is also composed of a linear layer followed by a softmax to solve the specific NLP task (for example, predict the next word in a sentence).
BERT stands for Bidirectional Encoder Representations from Transformers, so, as the name suggests, it is a way of learning representations of a language that uses a transformer, specifically, the encoder part of the transformer.
BERT is a language model, i.e. it represents the statistical relationships of the words in a language, i.e. which words are more likely to come after another word and stuff like that. Hence the part Representations in its name, Bidirectional Encoder Representations from Transformers.
BERT can be trained in an unsupervised way for representation learning, and then we can fine-tune BERT on the so-called downstream tasks in a supervised fashion (i.e. transfer learning). There are pre-trained versions of BERT that can be already fine-tuned (e.g. this one) and used to solve your specific supervised learning task. You can play with this TensorFlow tutorial to use a pre-trained BERT model.
On the other hand, the original transformer was not originally conceived to be a language model, but to solve sequence transduction tasks (i.e. converting one sequence to another, such as machine translation) without recurrent connections (or convolutions) but only attention.
BERT is only an encoder, while the original transformer is composed of an encoder and decoder. Given that BERT uses an encoder that is very similar to the original encoder of the transformer, we can say that BERT is a transformer-based model. So, BERT does not use recurrent connections, but only attention and feed-forward layers. There are other transformer-based neural networks that use only the decoder part of the transformer, for example, the GPT model.
BERT uses different hyper-parameters than the ones used in Attention is all you need to achieve the best performance. For example, it uses 12 and 16 "attention heads" (please, read the transformer paper to know more about these "attention heads") rather than 8 (although in the original transformer paper the authors experimented with a different number of heads).
BERT also uses segment embeddings, while the original transformer only uses word embeddings and positional encodings.
There are probably other small differences that I missed, but, after having read the paper Attention is all you need and quickly read some parts of the BERT paper, these seem to be the main differences.
Although I never used them, I would say that you want to use BERT whenever you want to solve an NLP task in a supervised fashion, but your labeled training dataset is not big enough to achieve good performance. In that case, you start with a pre-trained BERT model, then fine-tune it with your small labeled dataset. You probably need to add specific layers to BERT to solve your task.
