# Understanding how continuous bag of words method learns embedded representations

I'm reading notes on word vectors here. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding:

If we have a vocabulary $$V$$, the naive way to represent words in it would be via one-hot-encoding, or in other words, as basis vectors of $$R^{|V|}$$ - say $$e_1, e_2,\ldots,e_{|V|}$$. We want to map these to $$\mathbb{R}^n$$, via some linear transformation such that the images of similar words (more precisely, the images of basis vectors corresponding to similar words) have higher inner products. Assuming the matrix representation of the linear transformation given the standard basis of $$\mathbb{R}^{|V|}$$ is denoted by $$\mathcal{V}$$, then the "embedding" of the $$i$$-th vocab word (i.e. the image of the corresponding basis vector $$e_i$$ of $$V$$) is given by $$\mathcal{V}e_i$$.

Now suppose we have a context "The cat ____ over a", CBoW seeks to find a word that would fit into this context. Let the words "the", "cat", "over", "a" be denoted (in the space $$V$$) by $$x_{i_1},x_{i_2},x_{i_3},x_{i_4}$$ respectively. We take the image of their linear combination (in particular, their average): $$\hat v=\mathcal{V}\bigg(\frac{x_{i_1}+x_{i_2}+x_{i_3}+x_{i_4}}{4}\bigg)$$

We then map $$\hat v$$ back from $$\mathbb{R}^n$$ to $$\mathbb{R}^{|V|}$$ via another linear mapping whose matrix representation is $$\mathcal{U}$$: $$z=\mathcal{U}\hat v$$

Then we turn this score vector $$z$$ into softmax probabilities $$\hat y=softmax(z)$$ and compare it to the basis vector corresponding to the actual word, say $$e_c$$. For example, $$e_c$$ could be the basis vector corresponding to "jumped".

Here's my interpretation of what this procedure is trying to do: given a context, we're trying to learn maps $$\mathcal{U}$$ and $$\mathcal{V}$$ such that given a context like "the cat ____ over a", the model should give a high score to words like "jumped" or "leaped", etc. Not just that - but "similar" contexts should also give rise to high scores for "jumped", "leaped", etc. For example, given a context "that dog ____ above this" wherein "that", "dog", "above", "this" are represented by $$x_{j_1},x_{j_2},x_{j_3},x_{j_4}$$, let the image of their average be

$$\hat w=\mathcal{V}\bigg(\frac{x_{j_1}+x_{j_2}+x_{j_3}+x_{j_4}}{4}\bigg)$$

This gets mapped to a score vector $$z'=\mathcal{U}\hat w$$. Ideally, both score vectors $$z$$ and $$z'$$ should have similarly high magnitudes in their components corresponding to similar words "jumped" and "leaped".

Now to the questions:

We create two matrices, $$\mathcal{V} \in \mathbb{R}^{n\times |V|}$$ and $$\mathcal{U} \in \mathbb{R}^{|V|\times n}$$, where $$n$$ is an arbitrary size which defines the size of our embedding space. $$\mathcal{V}$$ is the input word matrix such that the $$i$$-th column of $$\mathcal{V}$$ is the $$n$$-dimensional embedded vector for word $$w_i$$ when it is an input to this model. We denote this $$n\times 1$$ vector as $$v_i$$. Similarly, $$\mathcal{U}$$ is the output word matrix. The $$j$$-th row of $$\mathcal{U}$$ is an $$n$$-dimensional embedded vector for word $$w_j$$ when it is an output of the model. We denote this row of $$\mathcal{U}$$ as $$u_j$$.

1. How does minimizing the cross-entropy loss between $$e_c$$ and $$\hat y$$ ensure that basis vectors corresponding to similar words $$e_i$$ and $$e_j$$ are mapped to vectors in $$\mathbb{R}^n$$ that have high inner product? I'm not sure of the mechanism how the above procedure ensures that. In other words, how is it ensured that if words no. $$i_1$$ and $$i_2$$ are similar, then $$\langle v_{i_1}, v_{i_2}\rangle$$ and $$\langle u_{i_1}, u_{i_2}\rangle$$ have high values?

2. How does the above procedure ensure that linear combinations of words in similar contexts are mapped to "similar" images? Does that even happen? In the above description for example, do $$\hat v$$ and $$\hat w$$ corresponding to similar contexts also have a high inner product? If so, how is that ensured?

3. Maybe my linear algebra is rusty and this is a silly question, but from what I gather, the columns of $$\mathcal{V}$$ represent the images of OHE vectors (standard basis of $$V$$) in the standard basis of $$\mathbb{R}^n$$ - i.e. the embedded representation of vocab words. Also, the rows of $$\mathcal{U}$$ also somehow represent the embedded representation of vocab words in $$\mathbb{R}^n$$. It's not obvious to me why $$v_i=\mathcal{V}e_i$$ should be the same as or even similar to $$u_i$$. Again, how does the above procedure ensure that?

• Hi. You're asking several questions in this post. Ideally, you should ask just one question per post, so that to facilitate the life of potential answerers (who might know the answer to one question but not the others). Furthermore, long posts I think might discourage people like me (who don't have much time) from reading them. I suggest you ask just one question in this post and ask the other questions in their separate post. – nbro Aug 13 '19 at 16:37
• @nbro: Thanks for the tip! I'll try to see how to reformulate the above question and break it down into parts. – Shirish Kulhari Aug 14 '19 at 1:29