Why is an embedding of dimension 400 enough to represent 70000 words?

Question

I am learning PyTorch on Udacity. In lesson 8, section 11: Training the Model, the instructor writes:

Then I have my embedding and hidden dimension. The embedding dimension is just a smaller representation of my vocabulary of 70k words and I think any value between like 200 and 500 or so would work, here. I've chosen 400. Similarly, for our hidden dimension, I think 256 hidden features should be enough to distinguish between positive and negative reviews.

There are more than 70000 different words. How could those more than 70000 unique words be represented by just 400 embeddings? How does an embedding look like? Is it a number?

Moreover, why would 256 hidden features be enough?

Answer 1

The specific term you are looking for is "word embedding" and not just "embedding".

How to numerically represent textual data?

Neural networks (typically) require as inputs (and produce as outputs) numerical data (i.e. numbers, vectors, matrices, or higher-dimensional arrays). So, when processing textual data, we first need to encode (or convert) the text into a numerical representation. There are different ways to do it, such as

• one-hot encoding (in that case, if you have 70000 words, you would have sparse vectors with 70000 entries where only one of those entries is equal to $1$ and all other entries are $0$: see this article for more info)

• map each word to a number (in this case, you would have 70000 numbers, one for each word)

• word embeddings

Each of these representations has different benefits and drawbacks. For instance, if you map each word to a number, then you just need to keep track of $70000$ numbers. In the case of one-hot encoding or word embeddings, you will need more memory. However, nowadays, word embeddings are widely used in natural language processing/understanding/generation tasks (and given that your question is about word embeddings), so let me briefly describe them.

Word embeddings

There are different word embedding techniques (such as word2vec). However, they are all based on the same ideas

1. Words that are similar (or related) in meaning should be mapped to vectors (i.e. the "word embeddings") that are also similar in some sense (for instance, their cosine similarity should be high). For instance, the words "man" and "boy" should be mapped to vectors that are similar.

2. These word embeddings are learned (rather than hard-coded or manually specified) given the data

3. The size of the word embeddings is a hyper-parameter (this should answer your question!)

Hyper-parameters

To answer your question(s) more directly, the choice of the dimension of the embeddings or the number of "hidden features" (which are both hyper-parameters) was probably more or less arbitrary or based on the instructor's experience. In general, it is difficult to determine the optimal choice of any hyper-parameter. Sometimes you can just use numbers that other people have used in the past and have noticed that work "well enough". If you really want to find more appropriate values of the hyper-parameters, you could use some hyper-parameter optimization technique, such as Bayesian optimization or a simple grid search.

Further reading

You can find many resources online that explain the concept of "word embeddings" more in detail. For instance

• What Are Word Embeddings for Text? (by Jason Brownlee)
• Word embeddings (A TensorFlow article/tutorial on word embeddings).

Answer 2

I finally grasped the concept of word embedding. Thanks to @nbro, after reading the 2 articles s/he recommended

1. What Are Word Embeddings for Text? and

2. Word embeddings

the 1st article gives me a good idea about the big picture of the Word Embeddings; whereas the 2nd article is actually the one which clears my mind.

I am an visual person, I understand things better if I could see how the things, in this case the Word Embeddings, look like(This answered my 2nd questions).

After seeing this image, my 1st question is answered and I realized that Word Embeddings is a 2 dimensional array where the number of rows of the array is decided by the number of unique words in your vocabulary and the columns/width is decided by yourself. Normally between 8 up to 1024 according to the 2nd article.

The columns/width within the course I am learning from is called embedding_dim, which I found hard to comprehend. Since each word embedding is a vector (this answered my 3rd question), for example the cat is [1.2, -0.1, 4.3, 3.2], and the vector is a meta concept for me which is easy to understand, I would like to call the embedding_dim : embedding_vector_width or embedding_vector_length.

For the 256 hidden features, how many of them would be enough, I think it's the same concept of how to figure out how many embedding_vector_width should be.