Word2Vec is an algorithm that generates word embeddings using logistic regression. Positive and negative examples are needed for logistic regression.

Word2Vec uses the neighboring words that are within the size of the context window as positive examples and noise words as negative examples.

I have a doubt regarding the selection of negative examples. The procedure, I am following, for selecting negative examples is taken from p20 of Vector Semantics and Embeddings

It is mentioned that random sampling is used to select negative examples

A noise word is a random word from the lexicon, constrained not to be the target word $w$.

But again, it is also mentioned that noise words are selected according to their weighted uni-gram frequencies

The noise words are chosen according to their weighted unigram frequency $p_{\alpha}(w)$, where $\alpha$ is a weight.

The same material is saying two contradictory procedures in selecting the noise words. Which one is true? Are they both alternative approaches?

How do we select a noise word based on its weighted uni-gram frequency? Do we pick the words that have the highest $p_{\alpha}(w)$?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.