# Transformer Language Model generating meaningless text

I currently learning on Transformers, so check my understanding I tried implementing a small transformer-based language model and compare it to RNN based language model. Here's the code for transformer. I'm using PyTorch inbuilt layer for Transformer Encoder

class TransformerLM_1(nn.Module):

def __init__(self, head, vocab_size, embedding_size, dropout = 0.1, device = 'cpu',
pad_idx = 0, start_idx = 1, end_idx = 2, unk_idx = 3):

super(TransformerLM_1, self).__init__()

self.embedding_size = embedding_size
self.vocab_size = vocab_size
self.device = device
self.postional_encoding = PostionalEncoding(embedding_size, device)
self.out_linear = nn.Linear(self.embedding_size, vocab_size)
self.dropout = dropout
self.start_idx = start_idx
self.end_idx = end_idx
self.unk_idx = unk_idx
self.device = device

mask = (torch.triu(torch.ones(src_sz, src_sz)) == 1).transpose(0, 1)

def forward(self, x):
dec_in = x.clone()[:, :-1]
src = self.embed(dec_in)
src = self.postional_encoding(src)
src = src.transpose(0,1)
out = self.out_linear(transformer_out)
return out


I'm using teacher forcing to make it converge faster. From what I saw from the results, the text generated by the RNN model is better than transformer's.

Here is sample generated text with the expected

Expected: you had to have been blind not to see the scenario there for what it was and is and will continue to be for months and even years a part of south carolina that has sustained a blow that the red cross expects will cost that organization alone some $$n million  Predicted: some  been the been  not be$$ the total has was the may has <unk> the that that be to the <unk> the

Expected: citicorp and chase are attempting to put together a new lower bid <eos>
Predicted: a are <unk> carries n't to the together with <unk> jersey than

Expected: it ' s amazing the amount of money that goes up their nose out to the dog track or to the tables in las vegas mr . katz says <eos>
Predicted: <unk> ' s <unk> comeback money of the in mr to their <unk> and of <unk> <unk> or or <unk> the money

Expected: moreover while asian and middle eastern investors <unk> gold and help <unk> its price silver does n't have the same <unk> dealers say <eos>
Predicted: the production the routes <unk> of its

Expected: a board of control spokesman said the board had not seen the claim and declined to comment <eos>
Predicted: the board said declined of said

Expected: property capital trust said it dropped its plan to liquidate because it was n't able to realize the value it had expected <eos>
Predicted: the claims markets said its was n <unk> to sell insolvent of was n't disclosed to sell its plan

Expected: similarly honda motor co . ' s sales are so brisk that workers <unk> they have n't had a saturday off in years despite the government ' s encouragement of more leisure activity <eos>
Predicted: the honda ' credit . s s <unk>

Expected: we expect a big market in the future so in the long term it will be profitable <eos>
Predicted: it can it <unk> board

Expected: u . k . composite or <unk> insurers which some equity analysts said might be heavily hit by the earthquake disaster helped support the london market by showing only narrow losses in early trading <eos>
Predicted: the . s . s trading sell said which <unk> traders market said the be able in the the earthquake

Expected: this will require us to define and <unk> what is necessary or appropriate care <eos>
Predicted: <unk> is be the \$ <unk> <unk> <unk> <unk> is the to <unk> and or


As you can see Transformer fails to grasp grammar compared to RNN. Is there anything wrong with my understanding?

EDIT

This is one example that caught my eye

Expected: also the big board met with angry stock specialists <eos>
Predicted: also met specialists board met the stock big with after


Most of the words predicted have is from the expected but in a different order. I have read that transformers are permutation invariant which is the reason why we include positional encoding with the word embedding.

• I may be asking very rudimentary question. Is the model getting pretrained with masked tokens prediction task ? If so, then do you think, it can be used for sentence generation ? Sep 29 '20 at 11:18
• No, it has not been pretrained with masked tokens. I'm following the tutorial given in pytorch documentation. pytorch.org/tutorials/beginner/transformer_tutorial.html. Also, I have tried masking random tokens during training, but the results are of the same nature. The sentences generated doesn't make any sense. Sep 29 '20 at 13:41