I'm using this source to understand the skip gram model. Let's say the context size is $4$ ($2$ context words on each side of the target word). This image illustrates how training examples are generated:

enter image description here

and this is in agreement with a few other sources and SE answers I've looked up. I'm also reading these notes which suggest the following architecture for skip gram:

enter image description here

This architecture suggests that a single training example must have the target word and all the context words in the window (e.g. (quick, brown, jumps, over, fox). The loss function as described in the lecture is (here the target word is indexed as $0$):

$$J=-\log P(w_{-2}, w_{-1}, w_1, w_2 | w_0) \\=-\log \prod_{i=-2,i\neq 0}^2 P(u_i | v_0) \\=-\log \prod_{i=-2,i\neq 0}^2 \frac{\exp(u_i^Tv_0)}{\sum_{k=1}^V\exp(u_k^Tv_0)}$$

Assuming the window here is $w_{-2}$: "quick", $w_{-1}$: "brown", $w_1$: "jumps", $w_2$: "over", with target word $w_0$: "fox",

  1. Is the neural network architecture correct? Based on the first link I posted, each training example has only one context word and so the last layer should only have $V$ nodes instead of $C\times V$.

  2. Accordingly, the loss function for each training example (say $(w_1, w_0)$) should be:

$$L(w_1, w_0) = -\log \frac{\exp(u_1^Tv_0)}{\sum_{k=1}^V\exp(u_k^Tv_0)}$$

Unlike a standard neural network, in which we form the cost function by taking the average of loss function over all training examples and then backpropagate on the basis of that, in this case we backpropagate with respect to the above loss function alone. We then move on to the next training example $w_2$, form a new loss function $L(w_2, w_0)$, backpropagate w.r.t. that, and so on. Is my understanding correct? Is the loss function as I described above correct? The other alternative I can think of is to form the cost function for that particular window by summing over loss functions for that window ($J=L(w_{-2},w_0)+L(w_{-1},w_0)+L(w_1,w_0)+L(w_2,w_0)$) and then doing backpropagation w.r.t. $J$.

  1. I've read that skip gram gives more weightage to nearby words rather than distant ones. I'm not sure why that's true. Given the window size of $2$ (say "quick brown fox jumps over"), it seems "quick" would be treated equivalently to "brown".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.