I was checking on sentence embeddings and stumbled across the BERT model which employs transformers.

I understand that BERT applies a WordPice tokenizer (e.g. working like https://keras.io/api/keras_nlp/tokenizers/word_piece_tokenizer/) and then passes the tokens through several (transformer) layers. If using the transformers library, the output of each hidden layer can be accessed easily as described here https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/ . For each token we can then obtain a word embedding and aggregate a sentence embedding by e.g. mean- or max-pooling over all word embeddings in a sentence.

On https://d2l.ai/chapter_natural-language-processing-pretraining/bert-pretraining.html , I found that BERT can be trained on e.g. WikiText-2 (https://blog.salesforceairesearch.com/the-wikitext-long-term-dependency-language-modeling-dataset/ ) but I do not see

  1. on which training task
  2. which loss function the original BERT model is trained?

This is curcial, since it determines what pattern the model picks up.

The last website states that the loss function is a cross-entropy loss. But I do not yet understand corss-entropy between what? What is the (X,y)-pairs used for training Bert?


2 Answers 2


I found the following in the original BERT paper https://arxiv.org/pdf/1810.04805v2.pdf : "The training loss is the sum of the mean masked LM [language model] likelihood and the mean next sentence prediction likelihood."

The original BERT model was trained to perform two natural language processing tasks: masked language modeling and next sentence prediction.

In masked language modeling, the model is given a sentence with some of the words masked out, and its task is to predict the original value of the masked words, based on the context provided by the rest of the sentence. This task is useful for learning contextual representations of words, because the model has to consider the full context in which the word appears in order to make a prediction.

In next sentence prediction, the model is given a pair of sentences and has to predict whether the second sentence is likely to follow the first one in a coherent text. This task helps the model learn about the relationships between sentences and how they fit together in a larger text.

The original BERT model was trained using a combination of these two tasks, with the masked language modeling task being the primary task and the next sentence prediction task being used as an auxiliary task. The model was trained to minimize the cross-entropy loss on these tasks, which is a common choice for classification tasks like these.


BERT is pre-trained using MLM and NSP tasks:

  • The MLM task is a type of self-supervised learning where some of the words in a sentence are randomly masked, and the model is trained to predict the masked words based on the surrounding context. For example, in the sentence "The cat is on the mat", a masked version could be "The [MASK] is on the mat", and the model is trained to predict the missing word (in this case, "cat").
  • The goal of the NSP task is to train BERT to understand the relationships between sentences. It helps BERT learn not just about individual words, but also about how sentences are connected in a broader context. The loss function used for the NSP task is a standard binary cross-entropy loss. It measures the difference between the predicted probability of "IsNext" and the actual label.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .