I currently learning on Transformers, so check my understanding I tried implementing a small transformer-based language model and compare it to RNN based language model. Here's the code for transformer. I'm using PyTorch inbuilt layer for Transformer Encoder

class TransformerLM_1(nn.Module):

    def __init__(self, head, vocab_size, embedding_size, dropout = 0.1, device = 'cpu', 
                 pad_idx = 0, start_idx = 1, end_idx = 2, unk_idx = 3):
        super(TransformerLM_1, self).__init__()
        self.head = head
        self.embedding_size = embedding_size
        self.vocab_size = vocab_size
        self.device = device
        self.embed = WordEmbedding(self.vocab_size, self.embedding_size, pad_idx)
        self.postional_encoding = PostionalEncoding(embedding_size, device)
        self.decoder = nn.TransformerEncoderLayer(self.embedding_size, self.head)
        self.out_linear = nn.Linear(self.embedding_size, vocab_size)
        self.dropout = dropout
        self.pad_idx = pad_idx
        self.start_idx = start_idx
        self.end_idx = end_idx
        self.unk_idx = unk_idx
        self.device = device

    def make_src_mask(self, src_sz):
        mask = (torch.triu(torch.ones(src_sz, src_sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, 10e-20).masked_fill(mask == 1, float(0.0))
        mask = mask.to(self.device)
        return mask

    def forward(self, x):
        dec_in = x.clone()[:, :-1]
        src_mask = self.make_src_mask(dec_in.size()[1])
        src = self.embed(dec_in)
        src = self.postional_encoding(src) 
        src = src.transpose(0,1)
        transformer_out = self.decoder(src, src_mask)
        out = self.out_linear(transformer_out)
        return out

I'm using teacher forcing to make it converge faster. From what I saw from the results, the text generated by the RNN model is better than transformer's.

Here is sample generated text with the expected

Expected: you had to have been blind not to see the scenario there for what it was and is and will continue to be for months and even years a part of south carolina that has sustained a blow that the red cross expects will cost that organization alone some $ n million <eos> 
Predicted: some <unk> been the been <unk> not be $ the total has was the may has <unk> the that that be to the <unk> the 

Expected: citicorp and chase are attempting to put together a new lower bid <eos> 
Predicted: a are <unk> carries n't to the together with <unk> jersey than 

Expected: it ' s amazing the amount of money that goes up their nose out to the dog track or to the tables in las vegas mr . katz says <eos> 
Predicted: <unk> ' s <unk> comeback money of the in mr to their <unk> and of <unk> <unk> or or <unk> the money 

Expected: moreover while asian and middle eastern investors <unk> gold and help <unk> its price silver does n't have the same <unk> dealers say <eos> 
Predicted: the production the routes <unk> of its 

Expected: a board of control spokesman said the board had not seen the claim and declined to comment <eos> 
Predicted: the board said declined of said 

Expected: property capital trust said it dropped its plan to liquidate because it was n't able to realize the value it had expected <eos> 
Predicted: the claims markets said its was n <unk> to sell insolvent of was n't disclosed to sell its plan 

Expected: similarly honda motor co . ' s sales are so brisk that workers <unk> they have n't had a saturday off in years despite the government ' s encouragement of more leisure activity <eos> 
Predicted: the honda ' credit . s s <unk> 

Expected: we expect a big market in the future so in the long term it will be profitable <eos> 
Predicted: it can it <unk> board 

Expected: u . k . composite or <unk> insurers which some equity analysts said might be heavily hit by the earthquake disaster helped support the london market by showing only narrow losses in early trading <eos> 
Predicted: the . s . s trading sell said which <unk> traders market said the be able in the the earthquake 

Expected: this will require us to define and <unk> what is necessary or appropriate care <eos> 
Predicted: <unk> is be the $ <unk> <unk> <unk> <unk> is the to <unk> and or 

As you can see Transformer fails to grasp grammar compared to RNN. Is there anything wrong with my understanding?


This is one example that caught my eye

Expected: also the big board met with angry stock specialists <eos> 
Predicted: also met specialists board met the stock big with after 

Most of the words predicted have is from the expected but in a different order. I have read that transformers are permutation invariant which is the reason why we include positional encoding with the word embedding.

  • $\begingroup$ I may be asking very rudimentary question. Is the model getting pretrained with masked tokens prediction task ? If so, then do you think, it can be used for sentence generation ? $\endgroup$ – Murugesh Sep 29 '20 at 11:18
  • $\begingroup$ No, it has not been pretrained with masked tokens. I'm following the tutorial given in pytorch documentation. pytorch.org/tutorials/beginner/transformer_tutorial.html. Also, I have tried masking random tokens during training, but the results are of the same nature. The sentences generated doesn't make any sense. $\endgroup$ – Hari Krishnan Sep 29 '20 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.