0
$\begingroup$

We are using the following notations, for this question, to calculate the probability values

\begin{array}{|c|c|} \hline \text{$w$} & \text{target word embedding vector} \\ \hline \text{$c$} & \text{context word embedding vector} \\ \hline \text{$L$} & \text{context window size} \\ \hline \text{$c_i$} & \text{$i^{th}$ context word} \\ \hline \text{$\sigma(x)$} & \dfrac{1}{1+e^{-x}} \\ \hline \text{$c.w$} & \text{dot product of $c$ and $w$} \\ \hline \end{array}

In Word2Vec, we learn embeddigs using logistic regression. In order to train logistic regression, there are two kinds of classes, one is positive (+) and other is negative (-).

The probability that the word $c$ is a real context word (positive class) for target word $w$ is

$$p(+/w, c) = \sigma(c.w) = \dfrac{1}{1+e^{-c.w}}$$

Since there are only two classes, we can imply the probability that the word $c$ is a not a real context word (negative class) for target word $w$ is

$$p(-/w, c) = \sigma(-c.w) = \dfrac{1}{1+e^{c.w}}$$

The first equation gives us the probability for one word, but there are many context words in the window. Skip-gram makes the simplifying assumption that all context words are independent, allowing us to just multiply their probabilities:

$$p(+/w, c_{1:L}) = \prod\limits_{i = 1}^{L} \sigma(-c.w)$$

My doubt is how does the RHS of the above equation is derived from the LHS?


I know that if all features $\{ X_1, X_2, X_3, \cdots, X_n \}$ are class conditionally independent with class $Y$, then

$$p(X_1, X_2, X_3, \cdots, X_n /Y) = p(X_1/Y)p(X_2/Y)p(X_3/Y) \cdots p(X_n/Y)$$

$$p(Y/ X_1, X_2, X_3, \cdots, X_n ) = \dfrac{p(X_1/Y)p(X_2/Y)p(X_3/Y) \cdots p(X_n/Y)}{p(X_1, X_2, X_3, \cdots, X_n)}$$

But I am not sure how to apply this and obtain the result I asked for.


For more information, one can check p19 of Vector Semantics andEmbeddings

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.