I currently learning on Transformers, so check my understanding I tried implementing a small transformer-based language model and compare it to RNN based language model. Here's the code for transformer. I'm using PyTorch inbuilt layer for Transformer Encoder
class TransformerLM_1(nn.Module): def __init__(self, head, vocab_size, embedding_size, dropout = 0.1, device = 'cpu', pad_idx = 0, start_idx = 1, end_idx = 2, unk_idx = 3): super(TransformerLM_1, self).__init__() self.head = head self.embedding_size = embedding_size self.vocab_size = vocab_size self.device = device self.embed = WordEmbedding(self.vocab_size, self.embedding_size, pad_idx) self.postional_encoding = PostionalEncoding(embedding_size, device) self.decoder = nn.TransformerEncoderLayer(self.embedding_size, self.head) self.out_linear = nn.Linear(self.embedding_size, vocab_size) self.dropout = dropout self.pad_idx = pad_idx self.start_idx = start_idx self.end_idx = end_idx self.unk_idx = unk_idx self.device = device def make_src_mask(self, src_sz): mask = (torch.triu(torch.ones(src_sz, src_sz)) == 1).transpose(0, 1) mask = mask.float().masked_fill(mask == 0, 10e-20).masked_fill(mask == 1, float(0.0)) mask = mask.to(self.device) return mask def forward(self, x): dec_in = x.clone()[:, :-1] src_mask = self.make_src_mask(dec_in.size()) src = self.embed(dec_in) src = self.postional_encoding(src) src = src.transpose(0,1) transformer_out = self.decoder(src, src_mask) out = self.out_linear(transformer_out) return out
I'm using teacher forcing to make it converge faster. From what I saw from the results, the text generated by the RNN model is better than transformer's.
Here is sample generated text with the expected
Expected: you had to have been blind not to see the scenario there for what it was and is and will continue to be for months and even years a part of south carolina that has sustained a blow that the red cross expects will cost that organization alone some $ n million <eos> Predicted: some <unk> been the been <unk> not be $ the total has was the may has <unk> the that that be to the <unk> the Expected: citicorp and chase are attempting to put together a new lower bid <eos> Predicted: a are <unk> carries n't to the together with <unk> jersey than Expected: it ' s amazing the amount of money that goes up their nose out to the dog track or to the tables in las vegas mr . katz says <eos> Predicted: <unk> ' s <unk> comeback money of the in mr to their <unk> and of <unk> <unk> or or <unk> the money Expected: moreover while asian and middle eastern investors <unk> gold and help <unk> its price silver does n't have the same <unk> dealers say <eos> Predicted: the production the routes <unk> of its Expected: a board of control spokesman said the board had not seen the claim and declined to comment <eos> Predicted: the board said declined of said Expected: property capital trust said it dropped its plan to liquidate because it was n't able to realize the value it had expected <eos> Predicted: the claims markets said its was n <unk> to sell insolvent of was n't disclosed to sell its plan Expected: similarly honda motor co . ' s sales are so brisk that workers <unk> they have n't had a saturday off in years despite the government ' s encouragement of more leisure activity <eos> Predicted: the honda ' credit . s s <unk> Expected: we expect a big market in the future so in the long term it will be profitable <eos> Predicted: it can it <unk> board Expected: u . k . composite or <unk> insurers which some equity analysts said might be heavily hit by the earthquake disaster helped support the london market by showing only narrow losses in early trading <eos> Predicted: the . s . s trading sell said which <unk> traders market said the be able in the the earthquake Expected: this will require us to define and <unk> what is necessary or appropriate care <eos> Predicted: <unk> is be the $ <unk> <unk> <unk> <unk> is the to <unk> and or
As you can see Transformer fails to grasp grammar compared to RNN. Is there anything wrong with my understanding?
This is one example that caught my eye
Expected: also the big board met with angry stock specialists <eos> Predicted: also met specialists board met the stock big with after
Most of the words predicted have is from the expected but in a different order. I have read that transformers are permutation invariant which is the reason why we include positional encoding with the word embedding.