How is it that word embedding layer (say word2vec) brings more insights to the network compared to a simple one hot encoded layer?
I understand how word embedding carry some semantic meaning, but it seems that this information would get "squashed" by the activation function, leaving only a scalar value and as many different vector could yield the same result, I would guess that the information is more or less lost.
Could anyone bring me insights as to why a network may utilize the information contained in a word embedding?