I'm reading "Recurrent neural network based language model" of Mikolov et al. (2010). Although the article is straight forward, I'm not sure how word embedding $w(t)$ is obtained:

enter image description here

The reason I wonder is that in the classic "A Neural Probabilistic Language Model" Bengio et al. (2003) - they used separate embedding vector for representing each word and it was somehow "semi-layer", meaning - it haven't contains non-linearity, but they did update word embeddings during the back-propagation.

In Mikolov approach though, I assume they used simple one-hot vector, where each feature represent presence of each word. If we represent that's way single word input (like was in the Mikolov's paper) - that vector become all-zeros except single one.

Is that correct?


1 Answer 1


Input vector contains two concatenated parts. The low part represents the current word:

word in time t encoded using 1-of-N coding [...] - size of vector x is equal to size of vocabulary V (this can be in practice 30 000-200 000) plus [...]

where, as you said, 1-of-N means (see here, 1-of-V):

If you have a fixed-size vocabulary of symbols with V members in total, each input symbol can be coded as a vector of size V with all zeros except for the element corresponding to the symbol's order in the vocabulary, which gets a 1.

The high part of the input vector represents the current context:

and previous context layer [...] Size of context (hidden) layer s is usually 30 - 500 hidden units.

For initialization, s(0) can be set to vector of small values, like 0.1

the article includes this expression:

x(t) = w(t) + s(t - 1)

that I think better to write as:

x(t) = w(t) || s(t - 1)

to made more visible the concatenation.

Finally, paper states some improvements that breaks the 1-to-N definition of the word (low) part, in order to reduce the size of the w vector:

we merge all words that occur less often than a threshold (in the training text) into a special rare token.

  • $\begingroup$ so x(t) = w(t) + s(t - 1) is in fact concatenation of 2 vectors, rather then addition? $\endgroup$
    – Ziemo
    Feb 15, 2018 at 16:26
  • $\begingroup$ yes, it is. "w" and "s" have very different sizes, no addition (sum componentwise) possible. $\endgroup$ Feb 15, 2018 at 16:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .