0
$\begingroup$

In word2vec, the task is to learn to predict which words are most likely to be near each other in some long corpus of text. For each word $c$ in the corpus, the model outputs the probability distribution $P(O=o|C=c)$ of how likely each other word $o$ in the vocabulary is to be within a certain number of words away from $c$. We call $c$ the "center word" and $o$ the "outside word".

We choose the softmax distribution as the output of our model: $$P(O=o|C=c) = \frac{\exp(\textbf{u}_{0}^{T} \textbf{v}_{c})}{\sum_{w \in \text{Vocab}} \exp(\textbf{u}_{w}^{T} \textbf{v}_c)}$$

where $\textbf{u}_0$ and $\textbf{v}_c$ are vectors that represent the outside and center words respectively.

Question. What do the vectors $\textbf{u}_0$ and $\textbf{v}_c$ look like? Are they just one-hot-encodings? Do we need to learn them too? Why is this useful?

$\endgroup$
1
$\begingroup$

No, the word vectors are not one-hot encodings. Yes, they are learned.

The purpose of the word2vec model is actually to learn dense, semantically meaningful encodings for words. That is, if your words are $d$-dimensional vectors, then each word's position in this vector space says something about what that word means. This is because word2vec learns to represent words in similar ways if they are frequently close together in your corpus. It implements the idea of distributional similarity.

The task of predicting an "outside word" given a "center word" accomplishes all of this in an indirect way.

A naive objective function to maximize for word2vec is $$J = \prod_{t=1}^L \prod_{-m \leq j \leq m\\ \quad j\neq 0} p(\textbf{u}_{t+j}|\textbf{v}_t)$$

where $L$ is the length of your corpus, $m$ is the "radius" from each center word you want to consider, $\textbf{u}_{t+j}$ is an outside word, and $\textbf{v}_t$ is a center word.

If we let $p(\textbf{u}_{t+j}|\textbf{v}_t)$ be the softmax distribution, then maximizing $J$ means maximizing the inner product $\textbf{u}_{t+j}^T\textbf{v}_t$ in the softmax's numerator. Maximizing that inner product means making center words as close as possible to their neighboring words, giving you some semantically meaningful word vectors to use in your downstream NLP tasks.

This lecture from Stanford's CS224N goes into more detail.

$\endgroup$
  • $\begingroup$ What is the OP talking about...Is it inference from the trained model? Or training the model? $\endgroup$ – DuttaA Mar 13 at 12:49
  • $\begingroup$ They're asking if training the model involves learning the "word vectors" u and v. These vectors are the parameters of the model. We don't do inference from word2vec, but the resulting word vectors are used to encode input text for some other NLP tasks. I'll edit the question to add more context. $\endgroup$ – Philip Raeisghasem Mar 13 at 19:51
  • $\begingroup$ But as far as I know word2vec is a type of autoencoder...And autoencoders are used for inference and other things...If my terminology is correct. $\endgroup$ – DuttaA Mar 13 at 20:32
  • 1
    $\begingroup$ Word2vec is like an autoencoder in that it learns useful encodings of inputs in an unsupervised way. It is not, however, a neural network. I also meant that we are not interested, ultimately, in the final output distribution of word2vec. We do not infer outputs of the model (distributions over likely nearby words) on new data. As in autoencoders, we may infer from a model that makes use of the learned encodings, but we do not typically infer from the model itself. Maybe I'm thinking of inference the wrong way, though. This is roughly my idea of inference. $\endgroup$ – Philip Raeisghasem Mar 13 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.