0
$\begingroup$

I am new to BERT and NLP and I am a little confused with tokenization and word embedding. My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use BertEmbedding for generating its corresponding word vectors of the tokens or I can train my own word2vec model to generate my word embedding while using BertTokenizer?

Pardon me if this question doesn't make any sense.

$\endgroup$
0
$\begingroup$

word2vec and BERT are completely different things. You don't need to use BERT Tokenizer for training word2vec model. But, if you want to go ahead. Here's a link which can help you.

$\endgroup$
4
  • $\begingroup$ Sorry, but I didn't ask that. My question was if I want to use BertTokenizer for a particular NLP task, say NER then is it mandatory to use BertEmbedding to generate the word embedding or I can train my own embeddings and use it along with BertTokenizer for that particular task (NER). $\endgroup$ Mar 13 '21 at 11:15
  • 1
    $\begingroup$ It is only mandatory if you are using BERT to create the embeddings. That's what I wrote in the answer also. You first tokenize then give it to BERT for generating embeddings. $\endgroup$ Mar 13 '21 at 11:28
  • $\begingroup$ Oh! thank you. Can you suggest to me any link or article for generating word embedding from BERT. $\endgroup$ Mar 13 '21 at 11:31
  • $\begingroup$ mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/… $\endgroup$ Mar 13 '21 at 11:35
0
$\begingroup$

Generally speaking, the power of BERT for applications like NER is that the authors (of whichever implementation you use) performed a large-scale pretraining effort to create the embeddings. You can then “fine-tune” those for your specific task using far less computation, but the rub is that you need to use the same tokenization scheme (I.e. the BERT Tokenizer) in order to have your input “fit” the existing embeddings. Intuitively, tokenization is mapping a word in your text to an index number. If the embedding was trained thinking that word number 42 is “cat” then things won’t work well if you tokenize differently and provide a 43 instead when “cat” pops up in your text.

Unless you’re training on a sparse language that hasn’t been well-represented by one of the public embeddings, the above is almost certainly your wisest approach. If, however you really want to train the BERT architecture on new embeddings, then you can technically use any embedding scheme you like.

The BERT Tokenizer uses subwords along with a few specific administrative tokens. If you were going to explore further, a byte pair encoder might be useful, especially if the language starts to beer away from eg English.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.