What do the vectors of the center and outside word look like in word2vec?

Question

In word2vec, the task is to learn to predict which words are most likely to be near each other in some long corpus of text. For each word $c$ in the corpus, the model outputs the probability distribution $P(O=o|C=c)$ of how likely each other word $o$ in the vocabulary is to be within a certain number of words away from $c$. We call $c$ the "center word" and $o$ the "outside word".

We choose the softmax distribution as the output of our model: $$P(O=o|C=c) = \frac{\exp(\textbf{u}_{0}^{T} \textbf{v}_{c})}{\sum_{w \in \text{Vocab}} \exp(\textbf{u}_{w}^{T} \textbf{v}_c)}$$

where $\textbf{u}_0$ and $\textbf{v}_c$ are vectors that represent the outside and center words respectively.

Question. What do the vectors $\textbf{u}_0$ and $\textbf{v}_c$ look like? Are they just one-hot-encodings? Do we need to learn them too? Why is this useful?

Answer 1

No, the word vectors are not one-hot encodings. Yes, they are learned.

The purpose of the word2vec model is actually to learn dense, semantically meaningful encodings for words. That is, if your words are $d$-dimensional vectors, then each word's position in this vector space says something about what that word means. This is because word2vec learns to represent words in similar ways if they are frequently close together in your corpus. It implements the idea of distributional similarity.

The task of predicting an "outside word" given a "center word" accomplishes all of this in an indirect way.

A naive objective function to maximize for word2vec is $$J = \prod_{t=1}^L \prod_{-m \leq j \leq m\\ \quad j\neq 0} p(\textbf{u}_{t+j}|\textbf{v}_t)$$

where $L$ is the length of your corpus, $m$ is the "radius" from each center word you want to consider, $\textbf{u}_{t+j}$ is an outside word, and $\textbf{v}_t$ is a center word.

If we let $p(\textbf{u}_{t+j}|\textbf{v}_t)$ be the softmax distribution, then maximizing $J$ means maximizing the inner product $\textbf{u}_{t+j}^T\textbf{v}_t$ in the softmax's numerator. Maximizing that inner product means making center words as close as possible to their neighboring words, giving you some semantically meaningful word vectors to use in your downstream NLP tasks.

This lecture from Stanford's CS224N goes into more detail.