I was checking on sentence embeddings and stumbled across the BERT model which employs transformers.

I understand that BERT applies a WordPice tokenizer (e.g. working like https://keras.io/api/keras_nlp/tokenizers/word_piece_tokenizer/) and then passes the tokens through several (transformer) layers. If using the transformers library, the output of each hidden layer can be accessed easily as described here https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/ . For each token we can then obtain a word embedding and aggregate a sentence embedding by e.g. mean- or max-pooling over all word embeddings in a sentence.

On https://d2l.ai/chapter_natural-language-processing-pretraining/bert-pretraining.html , I found that BERT can be trained on e.g. WikiText-2 (https://blog.salesforceairesearch.com/the-wikitext-long-term-dependency-language-modeling-dataset/ ) but I do not see

  1. on which training task
  2. which loss function the original BERT model is trained?

This is curcial, since it determines what pattern the model picks up.

The last website states that the loss function is a cross-entropy loss. But I do not yet understand corss-entropy between what? What is the (X,y)-pairs used for training Bert?


1 Answer 1


I found the following in the original BERT paper https://arxiv.org/pdf/1810.04805v2.pdf : "The training loss is the sum of the mean masked LM [language model] likelihood and the mean next sentence prediction likelihood."


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .