I am reading about RNN encoders. I came across the following line from this code. And I am facing difficulty in understanding the theoretical details regarding it.
emb = self.drop(self.encoder(input))
The input
is a tensor of shape $[32, 100]$. Here 32 is the batch size and 100 is the length of the sentence. Hundred elements are indices to the words (from the dictionary) that are used in the sentence. We can observe that the output emb
is later passed to the rnn (LSTM/GRU) layer.
output, hidden = self.rnn(emb, hidden)
So, to me, it looks like that self.encoder
is the necessary step while using the RNN encoder. So, I am interested in what it actually does.
When we see about self.encoder
, it is an Embedding layer. The description for this layer is as follows
A simple lookup table that stores embeddings of a fixed dictionary and size.
This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.
When we see about self.drop
, it randomly keeps zero in the embeddings.
During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.
The outputs for both self.encoder(input)
and self.drop(self.encoder(input))
are $[32, 100, 3000]$.
I have doubt(s) on the bolded parts of the description of the Embedding layer. The description is saying the Embedding layer uses/contains(?) a lookup table. The description says Embedding layer stores and retrieves word embeddings.
The doubts are
Generally, does an embedding layer calculate word embeddings or just store and retrieve them from the table? If it does not calculate them, then who will calculate the embeddings? If you can also comment on the specifics of PyTorch, I would appreciate it.
What exactly is an embedding layer? Is it a collection of neurons or any other?