How is the word embedding represented in the paper "Recurrent neural network based language model"?

Question

I'm reading "Recurrent neural network based language model" of Mikolov et al. (2010). Although the article is straight forward, I'm not sure how word embedding $w(t)$ is obtained:

The reason I wonder is that in the classic "A Neural Probabilistic Language Model" Bengio et al. (2003) - they used separate embedding vector for representing each word and it was somehow "semi-layer", meaning - it haven't contains non-linearity, but they did update word embeddings during the back-propagation.

In Mikolov approach though, I assume they used simple one-hot vector, where each feature represent presence of each word. If we represent that's way single word input (like was in the Mikolov's paper) - that vector become all-zeros except single one.

Is that correct?

Answer 1

Input vector contains two concatenated parts. The low part represents the current word:

word in time t encoded using 1-of-N coding [...] - size of vector x is equal to size of vocabulary V (this can be in practice 30 000-200 000) plus [...]

where, as you said, 1-of-N means (see here, 1-of-V):

If you have a fixed-size vocabulary of symbols with V members in total, each input symbol can be coded as a vector of size V with all zeros except for the element corresponding to the symbol's order in the vocabulary, which gets a 1.

The high part of the input vector represents the current context:

and previous context layer [...] Size of context (hidden) layer s is usually 30 - 500 hidden units.

For initialization, s(0) can be set to vector of small values, like 0.1

the article includes this expression:

x(t) = w(t) + s(t - 1)

that I think better to write as:

x(t) = w(t) || s(t - 1)

to made more visible the concatenation.

Finally, paper states some improvements that breaks the 1-to-N definition of the word (low) part, in order to reduce the size of the w vector:

we merge all words that occur less often than a threshold (in the training text) into a special rare token.