0
$\begingroup$

I am new to nlp field. I have some questions about word2vec embeddings. as I know they have a fixed size dictionary of vocabs. so definitely there some words which is not in that predefined dictionary of vocab, for i.e. we may get an special name (like people's name) which doesn't exist in the vocab and they are called out-of-vocabulary (OOV) words.

So how embedding and de-embedding for a translation language model work? I want the language model of the example to include the de-embedding part; as we know in classification language models we may only concern about the embedding part as the output is not an embedding vector, and does not need to be converted to a string(word)

I don't know what ways are used to embed those OOV words? I guess one way is to use some main word2vec and another complementary fastText for OOV words, or instead some complementary generic way of embedding like nn.embedding in pytorch. This won't make any problem in the classification models like sentiment analysis or Text classification.

Question 1: what are other methods that can be used for embedding an OOV word while we want to use word2vec?

but in the models which Their output is eventually gonna be a word (like translation or text summarization), I have some concerns as below:

Note we know the output of a translation language model is a sequence of embedding vectors to correspondent output word.

Note the way that we convert those embedding vectors to words, usually is by finding the closest embedding Vector in the embedding Matrix, either word2vec or fastText or de-embedder of nn.embedding. Also note this way eventually result, some Vector and from there, by its index, we would get a word.

as we have two embedding models, so when we are the step which have received the language model output embedding vector, we need first distinguish, this output embedding vector Should be de-embedded from which embedding model (word2vec or fastText)? note as I said eventually the next action (finding closest embedding) would result some word.

so I guess one way maybe is find the closest vectors for 2 embedding models, also we their similarity score to output embedding vector of the language model, and know to de-embed from which one.

question 2: so how this problem to distinguish to which embedding model to de-embed from is handled? I guess there should be more sophisticated methods in Practice so if there are please tell me?

question 3: do u know a model which have implemented my solutions (one for embedding by complementary embedding model, 2nd one distinguishing by similarity score)?

$\endgroup$
1
  • $\begingroup$ as I get its not probably the case and the tokenizers usually have the alphabet in the so if the word or its subwords are not in their vocab, they break it to alphabet which exists in the vocab $\endgroup$ Aug 21, 2023 at 15:00

1 Answer 1

1
$\begingroup$

I'm not sure I got 100% of the question, but word2vec is rather simple to understand so I'll give my two cents and if something is missing or not clear I'll edit the answer to improve it.

1 For OOV vectors, as you pointed out, one option is indeed to initialise some random embeddings. The nice aspect of this method is that these missing embeddings can be trained as extra parameters for the model while keeping the already trained vectors fixed. Another alternative is to get all zero vectors, meaning that we force the model to ignore certain words (it might be that these OOV are not of interest for specific use cases). Using vectors coming from other models is unlikely to work because of a problem called "vector alignment", i.e. embedding spaces from different model do not share the same learned geometric structure, meaning that two numerically equal vectors could be present in both spaces but be associated with totally different words.

2 Here I think there is a little misunderstanding going on regarding how word2vec works. First of all, word2vec is a training method for embedding, it does not necessarily refer only to pertained vector of fixed size. Specifically, the word2vec papers proposed two ways of training embeddings, continuous bag of words and skip-gram. The picture below show the difference, which I won't explain since there are already lot of answers about it in the community. The point I want to focus is that both approaches use simple classic supervised machine learning tasks. You have an input and you want to predict an output. In the original paper the desired output was a continuous vectors representation of words to leverage as features for further NLP tasks. This is different from encoding/decoding and the similarity is actually closer to transfer learning. It is also relevant to point out that you can get very good embedding also with classic matrix factorisation techniques, the problem is that these techniques can handle small amount of data, the deep learning formulation proposed in word2vec allow to train embeddings of ay dimension on any amount of data, hence the success of this approach.

3 As far as I'm aware there is no such model, and I doubt someone tried such approach for at least two reasons: 1 the same alignment problem already cited above, 2 the quick arise of transformers that happened not long after word2vec (attention is all you need was published in 2015, word2vec in 2013) which canalised all the attention (pun intended) in the NLP community to those new architectures.

enter image description here

$\endgroup$
1
  • $\begingroup$ thanks for the answer, but I have understood that OOV is not the case anymore because they dont let this to happpen. with having alphabet letters, even for other languages in the fixed vocab, they prevent OOV words to exist, because it would finally be breaked to some combination of subwords and letters. $\endgroup$ Sep 1, 2023 at 8:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .