3
$\begingroup$

I've been using Tensorflow and just started learning PyTorch. I was following the tutorial: https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-embeddings-tutorial-py

Where we try to create an n-gram language model. However, there's something I don't understand.

class NGramLanguageModeler(nn.Module):

    def __init__(self, vocab_size, embedding_dim, context_size):
        super(NGramLanguageModeler, self).__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.linear1 = nn.Linear(context_size * embedding_dim, 128)
        self.linear2 = nn.Linear(128, vocab_size)

at self.linear1 = nn.Linear(context_size * embedding_dim, 128) why did we multiply embedding_dim with context_size? Isn't the embedding_dim input size? So why do we multiply it by the context size?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

An n-gram language model is a language model trained with n context words. This means you're not feeding the model a single word but n. This is why the dimension of the input layer is "context_size * embedding_dim" or "n * embedding_dims"

Improve this answer
$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .