All Questions
Tagged with bert objective-functions
4 questions
0
votes
0
answers
519
views
What is the loss function used when pre-training BERT on MLM & NSP tasks?
I'm new to NLP and was reading through the 2019 BERT paper and am confused about the loss function used during pre-training.
As I understand it, the model is trained on the MLM and NSP tasks. The MLM ...
- 1
1
vote
0
answers
1k
views
What is MLM & NSP loss function
Two objective functions are used during the BERT language
model pretraining step.
The first one is masked language
model (MLM) that randomly masks
15% of the
input tokens and the objective is to ...
- 121
1
vote
0
answers
408
views
Left-to-Right vs Encoder-decoder Models
Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS)
Left-to-Right:
Auto-regressive, Left-to-right models, predict the ...
- 39
3
votes
1
answer
2k
views
Understanding how the loss was calculated for the SQuAD task in BERT paper
In the BERT paper, section 4.2 covers the SQuAD training.
From my understanding, there are two extra parameters trained, they are two vectors with the same dimension as the hidden size, so the same ...
- 197