We are using the following notations, for this question, to calculate the probability values

\begin{array}{|c|c|} \hline \text{$w$} & \text{target word embedding vector} \\ \hline \text{$c$} & \text{context word embedding vector} \\ \hline \text{$L$} & \text{context window size} \\ \hline \text{$c_i$} & \text{$i^{th}$ context word} \\ \hline \text{$\sigma(x)$} & \dfrac{1}{1+e^{-x}} \\ \hline \text{$c.w$} & \text{dot product of $c$ and $w$} \\ \hline \end{array}

In Word2Vec, we learn embeddigs using logistic regression. In order to train logistic regression, there are two kinds of classes, one is positive (+) and other is negative (-).

The probability that the word $c$ is a real context word (positive class) for target word $w$ is

$$p(+/w, c) = \sigma(c.w) = \dfrac{1}{1+e^{-c.w}}$$

Since there are only two classes, we can imply the probability that the word $c$ is a not a real context word (negative class) for target word $w$ is

$$p(-/w, c) = \sigma(-c.w) = \dfrac{1}{1+e^{c.w}}$$

The first equation gives us the probability for one word, but there are many context words in the window. Skip-gram makes the simplifying assumption that all context words are independent, allowing us to just multiply their probabilities:

$$p(+/w, c_{1:L}) = \prod\limits_{i = 1}^{L} \sigma(-c.w)$$

My doubt is how does the RHS of the above equation is derived from the LHS?

I know that if all features $\{ X_1, X_2, X_3, \cdots, X_n \}$ are class conditionally independent with class $Y$, then

$$p(X_1, X_2, X_3, \cdots, X_n /Y) = p(X_1/Y)p(X_2/Y)p(X_3/Y) \cdots p(X_n/Y)$$

$$p(Y/ X_1, X_2, X_3, \cdots, X_n ) = \dfrac{p(X_1/Y)p(X_2/Y)p(X_3/Y) \cdots p(X_n/Y)}{p(X_1, X_2, X_3, \cdots, X_n)}$$

But I am not sure how to apply this and obtain the result I asked for.

For more information, one can check p19 of Vector Semantics andEmbeddings


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.