3
$\begingroup$

When we use BERT embeddings for a classification task, would we get different embeddings everytime we pass the same text through the BERT architecture? If yes, is it the right way to use the embeddings as features? Ideally, while using any feature extraction technique, features values should be consistent. How do I handle this if we want BERT to be used as a feature extractor?

$\endgroup$
0
$\begingroup$

BERT is deterministic. There is no variation unless you parse your tokens differently in succeeding runs. Here is the original paper the model architecture is based off of Transformer Paper. Note that in every layer, the only operations used for the most part are matrix multiplications, concatenations, basic ops, and layer normalizations, all of which are deterministic.

$\endgroup$
  • $\begingroup$ Thats what I thought. But when I used the method 1 mentioned in the answers for the question here : stackoverflow.com/questions/55369821/… , I am getting different embeddings evrytime I run the code on same data with same tokenizer. $\endgroup$ – Srikant Jayaraman Jun 4 at 7:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.