Why is embedding important in NLP, and how does autoencoder work?

Question

People say embedding is necessary in NLP because if using just the word indices, the efficiency is not high as similar words are supposed to be related to each other. However, I still don't truly get it why.

The subword-based embedding (aka syllable-based embedding) is understandable, for example:

biology   --> bio-lo-gy
biologist --> bio-lo-gist

For the 2 words above, when turning them into syllable-based embeddings, it's good because the 2 words will be related to each other due to the sharing syllables: bio, and lo.

However, it's hard to understand the autoencoder, it turns an index value into vector, then feed these vectors to DNN. Autoencoder can turn vectors back to words too.

How does autoencoder make words related to each other?

Answer 1

The information you are probably missing is that word embeddings are learned on the basis of context. For example, you might try to predict a vector for a word from the wordvectors of the other words in the same sentence.

This way word vectors of words that occur in similar contexts will turn out to be similar. You can think of it as word vectors not encoding the word themselves but the contexts in which they are used. Of course ultimately that is the same.

Answer 2

The subword-based embedding is rather visual and easily understandable. However, the autoencoder embedding is what machines understand the componential meaning of words.

1) An autoencoder embedding layer can be trained together with other layers to fit with the relation of data in dataset.

2) Or the embedding layer can be kept unchanged as used as a function, required that the embedding layer must have been trained on a similar task like said in 1)

And as stated in the question, embedding is important in NLP as word indices don't show the full meanings of words, they must be separated into embedding values for better efficiency.

TensorFlow random uniform trainable embedding: https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding

TensorFlow utility class for creating subword-based encoder: https://www.tensorflow.org/datasets/api_docs/python/tfds/features/text/SubwordTextEncoder