I currently learning on Transformers, so check my understanding I tried implementing a small transformer-based language model and compare it to RNN based language model. Here's the code for transformer. I'm using PyTorch inbuilt layer for Transformer Encoder

class TransformerLM_1(nn.Module):

    def __init__(self, head, vocab_size, embedding_size, dropout = 0.1, device = 'cpu', 
                 pad_idx = 0, start_idx = 1, end_idx = 2, unk_idx = 3):
        super(TransformerLM_1, self).__init__()
        self.head = head
        self.embedding_size = embedding_size
        self.vocab_size = vocab_size
        self.device = device
        self.embed = WordEmbedding(self.vocab_size, self.embedding_size, pad_idx)
        self.postional_encoding = PostionalEncoding(embedding_size, device)
        self.decoder = nn.TransformerEncoderLayer(self.embedding_size, self.head)
        self.out_linear = nn.Linear(self.embedding_size, vocab_size)
        self.dropout = dropout
        self.pad_idx = pad_idx
        self.start_idx = start_idx
        self.end_idx = end_idx
        self.unk_idx = unk_idx
        self.device = device

    def make_src_mask(self, src_sz):
        mask = (torch.triu(torch.ones(src_sz, src_sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, 10e-20).masked_fill(mask == 1, float(0.0))
        mask = mask.to(self.device)
        return mask

    def forward(self, x):
        dec_in = x.clone()[:, :-1]
        src_mask = self.make_src_mask(dec_in.size()[1])
        src = self.embed(dec_in)
        src = self.postional_encoding(src) 
        src = src.transpose(0,1)
        transformer_out = self.decoder(src, src_mask)
        out = self.out_linear(transformer_out)
        return out

I'm using teacher forcing to make it converge faster. From what I saw from the results, the text generated by the RNN model is better than transformer's.

Here is sample generated text with the expected

Expected: you had to have been blind not to see the scenario there for what it was and is and will continue to be for months and even years a part of south carolina that has sustained a blow that the red cross expects will cost that organization alone some $ n million <eos> 
Predicted: some <unk> been the been <unk> not be $ the total has was the may has <unk> the that that be to the <unk> the 

Expected: citicorp and chase are attempting to put together a new lower bid <eos> 
Predicted: a are <unk> carries n't to the together with <unk> jersey than 

Expected: it ' s amazing the amount of money that goes up their nose out to the dog track or to the tables in las vegas mr . katz says <eos> 
Predicted: <unk> ' s <unk> comeback money of the in mr to their <unk> and of <unk> <unk> or or <unk> the money 

Expected: moreover while asian and middle eastern investors <unk> gold and help <unk> its price silver does n't have the same <unk> dealers say <eos> 
Predicted: the production the routes <unk> of its 

Expected: a board of control spokesman said the board had not seen the claim and declined to comment <eos> 
Predicted: the board said declined of said 

Expected: property capital trust said it dropped its plan to liquidate because it was n't able to realize the value it had expected <eos> 
Predicted: the claims markets said its was n <unk> to sell insolvent of was n't disclosed to sell its plan 

Expected: similarly honda motor co . ' s sales are so brisk that workers <unk> they have n't had a saturday off in years despite the government ' s encouragement of more leisure activity <eos> 
Predicted: the honda ' credit . s s <unk> 

Expected: we expect a big market in the future so in the long term it will be profitable <eos> 
Predicted: it can it <unk> board 

Expected: u . k . composite or <unk> insurers which some equity analysts said might be heavily hit by the earthquake disaster helped support the london market by showing only narrow losses in early trading <eos> 
Predicted: the . s . s trading sell said which <unk> traders market said the be able in the the earthquake 

Expected: this will require us to define and <unk> what is necessary or appropriate care <eos> 
Predicted: <unk> is be the $ <unk> <unk> <unk> <unk> is the to <unk> and or 

As you can see Transformer fails to grasp grammar compared to RNN. Is there anything wrong with my understanding?


This is one example that caught my eye

Expected: also the big board met with angry stock specialists <eos> 
Predicted: also met specialists board met the stock big with after 

Most of the words predicted have is from the expected but in a different order. I have read that transformers are permutation invariant which is the reason why we include positional encoding with the word embedding.

  • $\begingroup$ I may be asking very rudimentary question. Is the model getting pretrained with masked tokens prediction task ? If so, then do you think, it can be used for sentence generation ? $\endgroup$
    – Murugesh
    Commented Sep 29, 2020 at 11:18
  • $\begingroup$ No, it has not been pretrained with masked tokens. I'm following the tutorial given in pytorch documentation. pytorch.org/tutorials/beginner/transformer_tutorial.html. Also, I have tried masking random tokens during training, but the results are of the same nature. The sentences generated doesn't make any sense. $\endgroup$ Commented Sep 29, 2020 at 13:41

1 Answer 1


This is probably an issue of complete underfitting. How many training data do you use? What is your vocab size? What is your batch size and how many epochs did you train? Transformers always need more data than RNNs to reach good text quality.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .