I was checking on sentence embeddings and stumbled across the BERT model which employs transformers.
I understand that BERT applies a WordPice tokenizer (e.g. working like https://keras.io/api/keras_nlp/tokenizers/word_piece_tokenizer/) and then passes the tokens through several (transformer) layers. If using the transformers library, the output of each hidden layer can be accessed easily as described here https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/ . For each token we can then obtain a word embedding and aggregate a sentence embedding by e.g. mean- or max-pooling over all word embeddings in a sentence.
On https://d2l.ai/chapter_natural-language-processing-pretraining/bert-pretraining.html , I found that BERT can be trained on e.g. WikiText-2 (https://blog.salesforceairesearch.com/the-wikitext-long-term-dependency-language-modeling-dataset/ ) but I do not see
- on which training task
- which loss function the original BERT model is trained?
This is curcial, since it determines what pattern the model picks up.
The last website states that the loss function is a cross-entropy loss. But I do not yet understand corss-entropy between what? What is the (X,y)-pairs used for training Bert?