What exactly is embedding layer used in RNN encoders?

Question

I am reading about RNN encoders. I came across the following line from this code. And I am facing difficulty in understanding the theoretical details regarding it.

emb = self.drop(self.encoder(input))

The input is a tensor of shape $[32, 100]$. Here 32 is the batch size and 100 is the length of the sentence. Hundred elements are indices to the words (from the dictionary) that are used in the sentence. We can observe that the output emb is later passed to the rnn (LSTM/GRU) layer.

output, hidden = self.rnn(emb, hidden)

So, to me, it looks like that self.encoder is the necessary step while using the RNN encoder. So, I am interested in what it actually does.

When we see about self.encoder, it is an Embedding layer. The description for this layer is as follows

A simple lookup table that stores embeddings of a fixed dictionary and size.

This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.

When we see about self.drop, it randomly keeps zero in the embeddings.

During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.

The outputs for both self.encoder(input) and self.drop(self.encoder(input)) are $[32, 100, 3000]$.

I have doubt(s) on the bolded parts of the description of the Embedding layer. The description is saying the Embedding layer uses/contains(?) a lookup table. The description says Embedding layer stores and retrieves word embeddings.

The doubts are

1. Generally, does an embedding layer calculate word embeddings or just store and retrieve them from the table? If it does not calculate them, then who will calculate the embeddings? If you can also comment on the specifics of PyTorch, I would appreciate it.

2. What exactly is an embedding layer? Is it a collection of neurons or any other?

Answer 1

If you look at the source code of PyTorch's Embedding layer, you can see that it defines a variable called self.weight as a Parameter, which is a subclass of the Tensor, i.e. something that can be changed by gradient descent (you can do that by setting the parameter requires_grad of the Parameter to True). In other words, the Embedding layer is not just a look-up table, but it's a layer where you have parameters (i.e. the embeddings, which are stored in self.weight) that can also be learnable. You can also initialize these embeddings (i.e. the self.weight parameter) from pre-trained ones using Embedding's method from_pretrained. In this case, you should set require_grad to False.

Generally, one can define an embedding layer $\mathcal{f}$ as a function that receives the raw inputs $\mathbf{i}$ (e.g. in the case of word embeddings, the raw inputs might be integers: one for each word) and transforms them to embeddings $\mathbf{e}$, which can be statically defined (e.g. from pre-trained embeddings or hardcoded), randomly initialized and/or learnable (during the training of the neural network). In other words, $f(\mathbf{i}) = \mathbf{e}$. So, that's why we pass $f(\mathbf{i})$, i.e. the embeddings, rather than $\mathbf{i}$.