Skip-Gram Model Training

Suppose we want to predict context words $$w_{i-h}, \dots, w_{i+h}$$ given a target word $$w_i$$ for a window size $$h$$ around the target word $$w_i$$. We can represent this as: $$p(w_{i-h}, \dots, w_{i+h}|w_i) = \prod_{-h \leq k \leq h, \ k \neq 0} p(w_{i+k}|w_i)$$ where we model the probabilities of a word $$u$$ given another word $$v$$ as $$p(u|v) = \frac{\exp(\left<\phi_u, \theta_v \right>)}{\sum_{u' \in W} \exp(\left<\phi_{u'}, \theta_v \right>)}$$ where $$\phi_u, \theta_v$$ are some vector representations for words $$u$$ and $$v$$ respectively and $$\left<\phi_u, \theta_v \right>$$ is the dot product between these vector representations (which represents some sort of similarity between the words) and $$W$$ is a matrix of all the words.

In Skip-Gram Negative Sampling, we want to learn the embeddings $$\phi_u, \theta_v$$ that maximize the following: $$\sum_{u \in W} \sum_{v \in W} n_{uv} \log \sigma(\left<\phi_u, \theta_v \right>) +k \mathbb{E}_{\bar{v}} \log \sigma(-\left<\phi_u, \theta_{\bar{v}} \right>)$$

Question. How exactly does this work? For example, suppose $$k=5$$, the target word $$w_i$$ is $$\text{apple}$$ and we want to find $$p(\text{pie}| \text{apple})$$. Let $$n_{uv} = 10$$ (number of times pie co-occurs with apple). Then we sample $$5$$ random words $$\bar{v}$$ that did not occur with $$\text{apple}$$ and whichever term in the sum is bigger is the one we predict? For example, if the first term in the sum is larger than the second term then we would predict that $$p(\text{pie}| \text{apple}) \approx 1$$? Otherwise we predict that $$p(\text{pie}| \text{apple}) \approx 0$$? Is this the correct intuition?

Source. Here at around the 10:05 mark.

Almost, but no. When you maximize that objective function, you do so by adjusting the parameters $$\phi$$ and $$\theta$$. After you're done with training, you can use your word embeddings for other NLP tasks. You don't, however, do any prediction directly from the skip-gram model.

To maximize the first term, co-occuring words must have large inner products. That is, they must be "similar". To maximize the second term**, the randomly sampled words must have a small inner product with $$\phi_u$$. That is, they must be "dissimilar" to $$\phi_u$$. Moving these word embeddings around in the vector space to make some words similar and others not is the only thing that happens during skip-gram training. 

** $$\sigma(-x)=1-\sigma(x)$$, so maximizing $$\sigma(-x)$$ is minimizing $$\sigma(x)$$

• But isn't the idea of using Skip-Gram to bypass the computational complexity of the log-likelihood maximization? So instead of predicting a word given another word, we are predicting whether certain words appear together or not (implicitly)? See 9:43 of the video. – naturalguy_12 Mar 20 '19 at 15:09
• No. But that idea is similar to one behind the GloVe method of creating word embeddings. Also, I'm not really a fan of the video you keep linking. Higher quality videos from Stanford's 224d class are here. – Philip Raeisghasem Mar 23 '19 at 9:00
• Why aren't you a fan of the video? – naturalguy_12 Mar 25 '19 at 16:28