When we use BERT embeddings for a classification task, would we get different embeddings every time we pass the same text through the BERT architecture? If yes, is it the right way to use the embeddings as features? Ideally, while using any feature extraction technique, features values should be consistent. How do I handle this if we want BERT to be used as a feature extractor?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
BERT is deterministic. There is no variation unless you parse your tokens differently in succeeding runs. Here is the original paper the model architecture is based off of Transformer Paper. Note that in every layer, the only operations used for the most part are matrix multiplications, concatenations, basic ops, and layer normalizations, all of which are deterministic.
-
$\begingroup$ Thats what I thought. But when I used the method 1 mentioned in the answers for the question here : stackoverflow.com/questions/55369821/… , I am getting different embeddings evrytime I run the code on same data with same tokenizer. $\endgroup$ Jun 4, 2019 at 7:25