In Transformers Encoders, from where do the Encoders get Input Embedding from?

So when a sentence is given to a transformer-based model it first tokenises the sentence and each token is mapped with some integer in vocabulary. After that how does those tokens (integer) is converted to Input Embeddings?

If initial embeddings are obtained from some pre-trained model, how did that model was able to generate embeddings for a word?

enter image description here


2 Answers 2


Either by building embeddings yourself or loading pretrained embeddings.

For building yourself, this is typically done with an auto-regressive model. It can be as simple as creating numeric representations of all of the words in a corpus and generating random embeddings of whatever dimensionality you choose that are randomly initialized. The auto-regressive process could be to process your corpus word by word with the objective of predicting the next word in the corpus with some selected window size.

Alternatively, you could take your corpus as a chunk of any arbitrary size you choose, representing it as the equivalent vector made up of the word numbers (which is ultimately converted to a vector of embeddings for the words). Your training process would randomly (or progressively) exclude one of the words in the vector of embeddings with the objective of predicting the missing word.

Over time, the embeddings emerge.

  • $\begingroup$ It's bit confusing. Would be helpful if I can see explanation with some examples 🙈 $\endgroup$
    – Swastik
    Commented Oct 13, 2023 at 17:45
  • 1
    $\begingroup$ embeddings are just learned weights like every other parameter pytorch.org/docs/stable/generated/torch.nn.Embedding.html $\endgroup$
    – Karl
    Commented Oct 13, 2023 at 20:27
  • $\begingroup$ @Karl so before training a transformer model, we first need to train another model to get word to embedding? $\endgroup$
    – Swastik
    Commented Oct 14, 2023 at 4:23
  • $\begingroup$ like I said, embeddings are just learned weights like every other parameter. you train them at the same time as every other parameter. there's nothing special about the embedding weights. $\endgroup$
    – Karl
    Commented Oct 14, 2023 at 10:44
  • $\begingroup$ Ok let me be more clear. Given an input sentence, let's say below steps happens inputs -> tokens -> word embedding -> encoders. Here how is token to word embedding created? $\endgroup$
    – Swastik
    Commented Oct 14, 2023 at 12:33

Embeddings are essentially just a lookup table from integers to vector representations. These representations are learned during the pretraining process like any other parameter in the language model.

That is, they are initialized randomly then, as the model trains during pretraining (typically with some next-token prediction or denoising objective), gradients are propagated back to the embedding lookup table. Just as the regular parameters change to allow the model to predict tokens effectively, the token representations will also change to become more "meaningful" for the model.

If you further finetune the model, you use the existing pretrained parameters of the model, including the token embedding representations. Like any other parameter in the model, you can then choose to freeze or finetune the token embeddings as well.

You should just think of token embeddings like any other parameter in the model. Although there are other kinds of token embeddings (e.g., word2vec), in the context of large language model training, I'm not aware of any other initialization method other than what I described above, other than some niche scenarios (e.g., cross-lingual transfer: 1, 2, 3).

If you want a code example, you can take a look at the Embedding class in PyTorch.

  • $\begingroup$ How can next word prediction kind of architecture helps in generating initial word embeddings? $\endgroup$
    – Swastik
    Commented Oct 17, 2023 at 17:56
  • $\begingroup$ @Swastik They are learned through backpropagation like any other parameters in the model. As the model trains to be better at next token prediction, the embeddings need to become more meaningful, just as the parameters become more meaningful during training. $\endgroup$ Commented Oct 17, 2023 at 21:01
  • $\begingroup$ is there any code implementation which I can refer to understand more. Or if I want to build that from scratch just to learn how it works $\endgroup$
    – Swastik
    Commented Oct 18, 2023 at 1:20
  • $\begingroup$ @Swastik check out this tutorial: pytorch.org/tutorials/beginner/nlp/… $\endgroup$ Commented Oct 18, 2023 at 1:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .