I have recently researched on Bert structure. And the paper says we will mask some token at the input in 80%, 10% input be changed and 10% left remained. But I wonder if the mask token in the input be masked in attention of encoders layer? In that case, the input token of padding token and mask token is the same (equal 0). is it fine?



You must log in to answer this question.

Browse other questions tagged .