All Questions
Tagged with bert objective-functions
3 questions with no upvoted or accepted answers
1
vote
0
answers
1k
views
What is MLM & NSP loss function
Two objective functions are used during the BERT language
model pretraining step.
The first one is masked language
model (MLM) that randomly masks
15% of the
input tokens and the objective is to ...
- 121
1
vote
0
answers
409
views
Left-to-Right vs Encoder-decoder Models
Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS)
Left-to-Right:
Auto-regressive, Left-to-right models, predict the ...
- 39
0
votes
0
answers
519
views
What is the loss function used when pre-training BERT on MLM & NSP tasks?
I'm new to NLP and was reading through the 2019 BERT paper and am confused about the loss function used during pre-training.
As I understand it, the model is trained on the MLM and NSP tasks. The MLM ...
- 1