I'm new to NLP and was reading through the 2019 BERT paper and am confused about the loss function used during pre-training.
As I understand it, the model is trained on the MLM and NSP tasks. The MLM task is trained by passing the final hidden vectors corresponding to the mask tokens into a softmax function and then minimizing the cross entropy loss function between the softmax output and truth.
For the NSP task, I understand that the goal is to use the final hidden vector corresponding to the [CLS] token, and using that to determine whether the two "sentences" follow each other or not. Now the paper doesn't exactly say what loss function they use for this is, so I'm assuming they're doing something similar to the MLM case.
Now, in the appendix the authors mention that "The training loss is the sum of the mean MLM likelihood and the mean NSP likelihood". Given the relationship between log likelihood and cross entropy, this makes it seem like both MLK and NSP are used simultaneously during pre-training through a combined loss function.
I'm probably overthinking things, but I was wondering if any of you had a different interpretation of this. Specifically:
What is the loss function used when pre-training BERT on MLM & NSP? Also, what's your source?
I've read the paper start to end, as well as the transformers paper, searched forums, videos, and blogs, and my next stop is to dig into the code. I feel most explanations are either hand-wavy and it's unclear whether they're just guessing (like me) or if they actually know the answer.
