What is the difference between a language model and a word embedding?

Question

I am self-studying applications of deep learning on the NLP and machine translation.

I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score".

It appears to me that a language model is a way to predict the next word given its previous word. Word2vec is the similarity between two tokens. BLEU score is a way to measure the effectiveness of the language model.

Is my understanding correct? If not, can someone please point me to the right articles, paper, or any other online resources?

Answer 1

Simplified: Word Embeddings does not consider context, Language Models does.

For e.g Word2Vec, GloVe, or fastText, there exists one fixed vector per word.

Think of the following two sentences:

The fish ate the cat.

and

The cat ate the fish.

If you averaged their word embeddings, they would have the same vector, but, in reality, their meaning (semantic) is very different.

Then the concept of contextualized word embeddings arose with language models that do consider the context, and give different embeddings depending on the context.

Both word embeddings (e.g Word2Vec) and language models (e.g BERT) are ways of representing text, where language models capture more information and are considered state-of-the-art for representing natural language in a vectorized format.

BLEU score is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Which is not directly related to the difference between traditional word embeddings and contextualized word embeddings (aka language models).

Answer 2

A language model aims to estimate the probability of one or more words given the surrounding words. Given a sentence composed of $w_{1},...,w_{i-1},\_ , w_{i+1},..,w_{n}$, you can find which is the i-th missing word using a language model. In this way, you can estimate which is the most probable word using for example the conditional probability $P(w_i=w|w_1,…,w_n)$. An example of a simple language model is an n-gram where instead of conditioning on all previous words, you look only to the previous n words.

Word embeddings are a distributed representation of a word. Instead of using an index or a one-hot encoding to represent a word, a dense vector is used. If two words have similar embeddings then these words share some properties. These properties are driven by the way embeddings are constructed, for example in word2vec two words with similar embeddings are two words that often appear in the same context, which is not to say they have the same meaning. Sometimes words with opposite meanings can have similar embeddings just because they are placed within the same sentences/contexts.

The BLEU score is a way to quantify the translation quality of an automatic translation. The score aims to look at how different model translation is to human translation.