When to convert data to word embeddings in NLP

Question

When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings.

However, assuming the embeddings are pre-trained and frozen, there is another option. We could simply preprocess the training data prior to giving it to the model so that it is already converted to the embeddings. This will speed up training, since this conversion need only be performed once, as opposed to on the fly for each epoch.

Thus, the second option seems better. But the first choice seems more common. Assuming the embeddings are pre-trained and frozen, is there a reason I might choose the first option over the second?

Answer 1

If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than as a rank-4 tensor of [batches, documents, indices, vectors]. In this case, while the embedding is O(1) wherever you put it, it's more efficient to do so as part of the graph.

Answer 2

There are multiple ways to get word embedding from a corpus.

• Count Vectorizer: You can use the CountVectorizer() from sklearn.feature_extraction.text and then use the fit_transform() if the corpus has been converted into a list of sentences
• TF-IDF Vectorizer: You can use the TfidfVectorizer from sklearn.feature_extraction.text and then again use the fit_transform() on a list of sentences
• word2vec: You can make a word2vec model from gensim.models by using word2vec.Word2vec.

Answer 3

Assuming that the dictionary of the words, that your model comes up with, is a subset of the pretrained embeddings, for example of Google's pretrained word2vec, then it is maybe a better option following these embeddings, if your model can handle that size of dimension.

However, sometimes that would not always be the best solution, taking into account the nature of the problem. For example, if you are trying to use NLP on medical texts that contain rare and special words, then maybe you should use your embedding layer, assuming that you have an adequate data size, or both of them. That is just a thought of mine. For sure, there can be several other use cases which should propose the embedding layer.