Transformer Language Model generating meaningless text

Question

I currently learning on Transformers, so check my understanding I tried implementing a small transformer-based language model and compare it to RNN based language model. Here's the code for transformer. I'm using PyTorch inbuilt layer for Transformer Encoder

class TransformerLM_1(nn.Module):

    def __init__(self, head, vocab_size, embedding_size, dropout = 0.1, device = 'cpu', 
                 pad_idx = 0, start_idx = 1, end_idx = 2, unk_idx = 3):
      
        super(TransformerLM_1, self).__init__()
      
        self.head = head
        self.embedding_size = embedding_size
        self.vocab_size = vocab_size
        self.device = device
        self.embed = WordEmbedding(self.vocab_size, self.embedding_size, pad_idx)
        self.postional_encoding = PostionalEncoding(embedding_size, device)
        self.decoder = nn.TransformerEncoderLayer(self.embedding_size, self.head)
        self.out_linear = nn.Linear(self.embedding_size, vocab_size)
        self.dropout = dropout
        self.pad_idx = pad_idx
        self.start_idx = start_idx
        self.end_idx = end_idx
        self.unk_idx = unk_idx
        self.device = device

    
    def make_src_mask(self, src_sz):
        mask = (torch.triu(torch.ones(src_sz, src_sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, 10e-20).masked_fill(mask == 1, float(0.0))
        mask = mask.to(self.device)
        return mask

    def forward(self, x):
        dec_in = x.clone()[:, :-1]
        src_mask = self.make_src_mask(dec_in.size()[1])
        src = self.embed(dec_in)
        src = self.postional_encoding(src) 
        src = src.transpose(0,1)
        transformer_out = self.decoder(src, src_mask)
        out = self.out_linear(transformer_out)
        return out

I'm using teacher forcing to make it converge faster. From what I saw from the results, the text generated by the RNN model is better than transformer's.

Here is sample generated text with the expected

Expected: you had to have been blind not to see the scenario there for what it was and is and will continue to be for months and even years a part of south carolina that has sustained a blow that the red cross expects will cost that organization alone some $ n million <eos> 
Predicted: some <unk> been the been <unk> not be $ the total has was the may has <unk> the that that be to the <unk> the 

Expected: citicorp and chase are attempting to put together a new lower bid <eos> 
Predicted: a are <unk> carries n't to the together with <unk> jersey than 

Expected: it ' s amazing the amount of money that goes up their nose out to the dog track or to the tables in las vegas mr . katz says <eos> 
Predicted: <unk> ' s <unk> comeback money of the in mr to their <unk> and of <unk> <unk> or or <unk> the money 

Expected: moreover while asian and middle eastern investors <unk> gold and help <unk> its price silver does n't have the same <unk> dealers say <eos> 
Predicted: the production the routes <unk> of its 

Expected: a board of control spokesman said the board had not seen the claim and declined to comment <eos> 
Predicted: the board said declined of said 

Expected: property capital trust said it dropped its plan to liquidate because it was n't able to realize the value it had expected <eos> 
Predicted: the claims markets said its was n <unk> to sell insolvent of was n't disclosed to sell its plan 

Expected: similarly honda motor co . ' s sales are so brisk that workers <unk> they have n't had a saturday off in years despite the government ' s encouragement of more leisure activity <eos> 
Predicted: the honda ' credit . s s <unk> 

Expected: we expect a big market in the future so in the long term it will be profitable <eos> 
Predicted: it can it <unk> board 

Expected: u . k . composite or <unk> insurers which some equity analysts said might be heavily hit by the earthquake disaster helped support the london market by showing only narrow losses in early trading <eos> 
Predicted: the . s . s trading sell said which <unk> traders market said the be able in the the earthquake 

Expected: this will require us to define and <unk> what is necessary or appropriate care <eos> 
Predicted: <unk> is be the $ <unk> <unk> <unk> <unk> is the to <unk> and or 

As you can see Transformer fails to grasp grammar compared to RNN. Is there anything wrong with my understanding?

EDIT

This is one example that caught my eye

Expected: also the big board met with angry stock specialists <eos> 
Predicted: also met specialists board met the stock big with after 

Most of the words predicted have is from the expected but in a different order. I have read that transformers are permutation invariant which is the reason why we include positional encoding with the word embedding.

Answer 1

This is probably an issue of complete underfitting. How many training data do you use? What is your vocab size? What is your batch size and how many epochs did you train? Transformers always need more data than RNNs to reach good text quality.