# What is the loss function and training task on which the original BERT model was trained

I was checking on sentence embeddings and stumbled across the BERT model which employs transformers.

I understand that BERT applies a WordPice tokenizer (e.g. working like https://keras.io/api/keras_nlp/tokenizers/word_piece_tokenizer/) and then passes the tokens through several (transformer) layers. If using the transformers library, the output of each hidden layer can be accessed easily as described here https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/ . For each token we can then obtain a word embedding and aggregate a sentence embedding by e.g. mean- or max-pooling over all word embeddings in a sentence.

On https://d2l.ai/chapter_natural-language-processing-pretraining/bert-pretraining.html , I found that BERT can be trained on e.g. WikiText-2 (https://blog.salesforceairesearch.com/the-wikitext-long-term-dependency-language-modeling-dataset/ ) but I do not see