Why does skip-gram uses linear maps for embedding?

Question

This question is inspired from Lilian Weng's blog here about skip-gram model, where she shows the model as multiplication via 2 matrices $W$ and $W'$ for embedding and word context matrix and $W$ takes in one hot encoded embedding into a 'better' embedding space.

My question is, why should we require a linear map for this task? As in, why should some linear map from one-hot encoded space be useful for embedding words densely, while preserving semantic information as much as possible (this is the definition of word 'better' as used above)?

For an overview of the model, see here:

Answer 1

Whether linear maps are "required" is an extremely difficult question -- I'm sure there's a number of ways to implement the general idea for the skip gram architecture. But this is more of a research question.

This might actually be the intention of your question, but there are definitely clear reasons why linear maps are an intuitive choice.

As discussed in the post, the goal is the convert the sparse one-hot representations into dense vectors. One intuition is that the meaning of words is determined by it's neighbors -- so skip-gram learns a model that can project one-hot representations into a lower dimensional representation such that that representation can be used to predict the held out word. This may remind you of autoencoders. Finally, linear maps are an extremely easy way to project vectors into different dimensions (it's just a matrix multiply!), making it an intuitive choice here.

In fact, reading the original paper, the mention:

The main observation from the previous section was that most of the complexity is caused by the non-linear hidden layer in the model. While this is what makes neural networks so attractive, we decided to explore simpler models that might not be able to represent the data as precisely as neural networks, but can possibly be trained on much more data efficiently.

i.e., linear maps are appealing because they're simple.