Intuition on how word embeddings bring information to a network

Question

How is it that word embedding layer (say word2vec) brings more insights to the network compared to a simple one hot encoded layer?

I understand how word embedding carry some semantic meaning, but it seems that this information would get "squashed" by the activation function, leaving only a scalar value and as many different vector could yield the same result, I would guess that the information is more or less lost.

Could anyone bring me insights as to why a network may utilize the information contained in a word embedding?

Answer 1

Shakespeare once said "A rose by any other name would smell as sweet" (Romeo and Juliet). Words are just labels we attach to ideas for convenience. By using one hot we remain tied to the letter sequence r,o,s,e, and some other structure must take on the responsibility of attaching the context of sweetness to it.

Word embeddings learn a multi-dimensional context. What exactly the context is of each dimension of the embedding is something of a mystery and simply emerges from the learning. The larger the number of dimensions, the greater the possibility that some combination of the dimensions will represent the sweetness context, but it might be quite hard to tease out.

So you can attach the idea of sweetness to one member of a one-hot structure, but it must necessarily be a part of a rules-based approach. Embeddings, when they are working well, will not need the rules.

Answer 2

Adding to Colin's answer; using word embedding tend to be much more robust that one-hot vectors. Consider the the following two sentences:

The desk has a book on it.

and

The table has a book on it.

These two sentences are almost identical in meaning. If we were to using word embeddings, the vectors 'desk' and 'table' would be very close together. The fact that these two sentences are similar becomes implicit with embeddings.

But if we were to use one-hot vectors, the distance between the two vectors would be the same distance between 'desk' and 'cat' or 'table' and 'book'. So now the network must learn that these sentences may entail the same thing on top of the original task.