I am currently trying to understand transformers.

To start, I read Attention Is All You Need and also this tutorial.

What makes me wonder is the word embedding used in the model. Is word2vec or GloVe being used? Are the word embeddings trained from scratch?

In the tutorial linked above, the transformer is implemented from scratch and nn.Embedding from pytorch is used for the embeddings. I looked up this function and didn't understand it well, but I tend to think that the embeddings are trained from scratch, right?

  • $\begingroup$ If you were to implement a Transformer or Word2Vec yourself with PyTorch, in both cases you will probably be using nn.Embedding. Agreed, the original paper is not clear on this. (The other parts of your question have already been answered well by you and in comments.) $\endgroup$
    – 0dB
    Jun 10 at 7:15

3 Answers 3


I have found a good answer in this blog post The Transformer: Attention Is All You Need:

we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some information about the word. We can do this using nn.Embedding in Pytorch, or, more generally speaking, by multiplying our one-hot vector with a learned weight matrix W.

There are two options for dealing with the Pytorch nn.Embedding weight matrix. One option is to initialize it with pre-trained embeddings and keep it fixed, in which case it’s really just a lookup table. Another option is to initialize it randomly, or with pre-trained embeddings, but keep it trainable. In that case the word representations will get refined and modified throughout training because the weight matrix will get refined and modified throughout training.

The Transformer uses a random initialization of the weight matrix and refines these weights during training – i.e. it learns its own word embeddings.

  • $\begingroup$ This response is confusing because it basically says it could be either trained or not trained. But I assume there is a single overall approach taken by the initial transformers paper, gpt-*, BERT, BLOOM, etc. The original Vaswani paper and GPT papers don't mention anything about an initial word embedding, nor do they mention anything about a trainable embedding matrix -- only trainable projections. So which one is it in those specific cases? Is the initial embedding trained or not? $\endgroup$
    – Paul
    Mar 1 at 15:46
  • $\begingroup$ When you set up a transformer you can choose if you want to train from scratch or with pre-trained embeddings, and for the latter, whether it should stay fixed or continue to be trained, just like the answer says. Embeddings and projections are the same concept, the first applied to tokens (non-numbers), the second to numbers, but sometimes you see the terms used synonymously. $\endgroup$
    – 0dB
    Jun 10 at 6:34

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings.

The embeddings are trained from scratch.

  • $\begingroup$ Maybe you should provide more details about the embeddings used by the transformers (apart from saying that they are learned from scratch). $\endgroup$
    – nbro
    Feb 5, 2021 at 21:11
  • $\begingroup$ Embeddings in Word2Vec are also trained from scratch $\endgroup$
    – Alex
    Feb 6, 2021 at 0:09
  • 2
    $\begingroup$ And how are they trained from scratch then? In the tutorials I have read yet that part is never mentioned. Every time just the attention part is focussed and the embedding part is skipped. $\endgroup$
    – Bert Gayus
    Feb 6, 2021 at 12:35
  • $\begingroup$ This answer is deceptive. After an input sequence has been tokenized, the transformer's first step will be to use a STATIC matrix to find that token's embedding. Bert Gayus' answer gives some insight into how that static matrix is trained. $\endgroup$ Jan 17 at 0:09
  • $\begingroup$ If you were to implement a Transformer or Word2Vec yourself with PyTorch, in both cases you will probably be using nn.Embedding. No, the embeddings are not necessarily trained from scratch, see my comment on the accepted answer. $\endgroup$
    – 0dB
    Jun 10 at 7:06

As "initial" word embeddings (those without any positional or context information for each word or sub word) are used from the very beginning It seems to me that someone has to provide a trained embedding for each word at the very beginning.

  • 1
    $\begingroup$ They can and will be trained from random starting values. $\endgroup$
    – 0dB
    May 15 at 18:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .