I am currently trying to understand transformers.

To start, I read Attention Is All You Need and also this tutorial.

What makes me wonder is the word embedding used in the model. Is word2vec or GloVe being used? Are the word embeddings trained from scratch?

In the tutorial linked above, the transformer is implemented from scratch and nn.Embedding from pytorch is used for the embeddings. I looked up this function and didn't understand it well, but I tend to think that the embeddings are trained from scratch, right?


3 Answers 3


I have found a good answer in this blog post The Transformer: Attention Is All You Need:

we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some information about the word. We can do this using nn.Embedding in Pytorch, or, more generally speaking, by multiplying our one-hot vector with a learned weight matrix W.

There are two options for dealing with the Pytorch nn.Embedding weight matrix. One option is to initialize it with pre-trained embeddings and keep it fixed, in which case it’s really just a lookup table. Another option is to initialize it randomly, or with pre-trained embeddings, but keep it trainable. In that case the word representations will get refined and modified throughout training because the weight matrix will get refined and modified throughout training.

The Transformer uses a random initialization of the weight matrix and refines these weights during training – i.e. it learns its own word embeddings.

  • 3
    $\begingroup$ Please, if you're quoting something from another source, prepend > to the excerpt you're quoting. See my edit to understand how to do it. $\endgroup$
    – nbro
    Feb 6, 2021 at 15:15

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings.

The embeddings are trained from scratch.

  • $\begingroup$ Maybe you should provide more details about the embeddings used by the transformers (apart from saying that they are learned from scratch). $\endgroup$
    – nbro
    Feb 5, 2021 at 21:11
  • $\begingroup$ Embeddings in Word2Vec are also trained from scratch $\endgroup$
    – Alex
    Feb 6, 2021 at 0:09
  • 1
    $\begingroup$ And how are they trained from scratch then? In the tutorials I have read yet that part is never mentioned. Every time just the attention part is focussed and the embedding part is skipped. $\endgroup$
    – Bert Gayus
    Feb 6, 2021 at 12:35

As "initial" word embeddings (those without any positional or context information for each word or sub word) are used from the very beginning It seems to me that someone has to provide a trained embedding for each word at the very beginning.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .