In his article word2vec Parameter Learning Explained Xin Rong says (page 7):

Each output is computed using the same hidden->output matrix: $$ p(w_{c,j} = w_{O.c}|w_I)=y_{c,j}=\frac{exp(u_{c,j})}{\sum_{j'=1}^{V}exp(u_{j'})} \ \ \ \ (25) $$

Looking into the word2vec source code I don’t see any “panels” or “output layers”. With this most of the terms in the equation above in totally unclear for me. Could you please help me realizing how this mathematical description intended to work (I understand the source code, but mathematical description is another question here).

Am I missing something here? With the description above from the article, since “the output layer panels share the same weights”, how the results on output panels (even if they existed could be different)?

  • $\begingroup$ The right hand side is called a softmax function, it turns a set of numbers, eg. [10, 3, 1, 0] into a probability distribution [0.99, 001, 0, 0] which always totals 1. It means the probability of the 1st token/word appearing next is 0.99, the probability of the 2nd token appearing next is 0.01, and there is no chance of the 3rd of 4th token appearing next. $\endgroup$
    – James
    Commented Jun 11 at 2:16
  • $\begingroup$ @James, thank you for the input. I know what Softmax is and more than that I know how hierarchical softmax works here. May be my wording of the question is vague, but I am concerned move with C-indexes here, since in the code which is described by this formula there are no panels at all. Meanwhile I noticed that the actual code uses "window" of words in the text so maybe the panels are "virtual" in the implementation. That is my question here. And if we have the same input word and same weights on the panels, how the outcome could be different. I know how it works in code, but the formula... $\endgroup$ Commented Jun 11 at 12:52
  • $\begingroup$ I skimmed through the pdf but the index notation seems a lot of work to wade into. Conceptually though as I understand it the skipgram is a simple tallying of surrounding words relative to the current word, eg. for sentence "am i missing something here", the skipgram for the word "missing" should read [0, 0.5, 0, 0.5, 0] for just this one sentence. The probabilities gets muddied where there are multiple occurences of the word "missing" throughout the learning text. $\endgroup$
    – James
    Commented Jun 11 at 17:37
  • $\begingroup$ @James, thanks again. And again, I know what is the skip-gram and how exactly it works. I was interested in one thing, how math describes the existing code to learn they way to cover things. The author of the paper here uses "panels" which are not represented explicitly in code which was my first concern. Then he says "the output layer panels share the same weights", if this is true than for the same input word they must produce the same output. Again, I know how this works (correctly in code), I am struggling with the "math" description of this. $\endgroup$ Commented Jun 11 at 21:26


You must log in to answer this question.

Browse other questions tagged .