I currently learning on Transformers, so check my understanding I tried implementing a small transformer-based language model and compare it to RNN based language model. Here's the code for transformer. I'm using PyTorch inbuilt layer for Transformer Encoder
class TransformerLM_1(nn.Module):
def __init__(self, head, vocab_size, embedding_size, dropout = 0.1, device = 'cpu',
pad_idx = 0, start_idx = 1, end_idx = 2, unk_idx = 3):
super(TransformerLM_1, self).__init__()
self.head = head
self.embedding_size = embedding_size
self.vocab_size = vocab_size
self.device = device
self.embed = WordEmbedding(self.vocab_size, self.embedding_size, pad_idx)
self.postional_encoding = PostionalEncoding(embedding_size, device)
self.decoder = nn.TransformerEncoderLayer(self.embedding_size, self.head)
self.out_linear = nn.Linear(self.embedding_size, vocab_size)
self.dropout = dropout
self.pad_idx = pad_idx
self.start_idx = start_idx
self.end_idx = end_idx
self.unk_idx = unk_idx
self.device = device
def make_src_mask(self, src_sz):
mask = (torch.triu(torch.ones(src_sz, src_sz)) == 1).transpose(0, 1)
mask = mask.float().masked_fill(mask == 0, 10e-20).masked_fill(mask == 1, float(0.0))
mask = mask.to(self.device)
return mask
def forward(self, x):
dec_in = x.clone()[:, :-1]
src_mask = self.make_src_mask(dec_in.size()[1])
src = self.embed(dec_in)
src = self.postional_encoding(src)
src = src.transpose(0,1)
transformer_out = self.decoder(src, src_mask)
out = self.out_linear(transformer_out)
return out
I'm using teacher forcing to make it converge faster. From what I saw from the results, the text generated by the RNN model is better than transformer's.
Here is sample generated text with the expected
Expected: you had to have been blind not to see the scenario there for what it was and is and will continue to be for months and even years a part of south carolina that has sustained a blow that the red cross expects will cost that organization alone some $ n million <eos>
Predicted: some <unk> been the been <unk> not be $ the total has was the may has <unk> the that that be to the <unk> the
Expected: citicorp and chase are attempting to put together a new lower bid <eos>
Predicted: a are <unk> carries n't to the together with <unk> jersey than
Expected: it ' s amazing the amount of money that goes up their nose out to the dog track or to the tables in las vegas mr . katz says <eos>
Predicted: <unk> ' s <unk> comeback money of the in mr to their <unk> and of <unk> <unk> or or <unk> the money
Expected: moreover while asian and middle eastern investors <unk> gold and help <unk> its price silver does n't have the same <unk> dealers say <eos>
Predicted: the production the routes <unk> of its
Expected: a board of control spokesman said the board had not seen the claim and declined to comment <eos>
Predicted: the board said declined of said
Expected: property capital trust said it dropped its plan to liquidate because it was n't able to realize the value it had expected <eos>
Predicted: the claims markets said its was n <unk> to sell insolvent of was n't disclosed to sell its plan
Expected: similarly honda motor co . ' s sales are so brisk that workers <unk> they have n't had a saturday off in years despite the government ' s encouragement of more leisure activity <eos>
Predicted: the honda ' credit . s s <unk>
Expected: we expect a big market in the future so in the long term it will be profitable <eos>
Predicted: it can it <unk> board
Expected: u . k . composite or <unk> insurers which some equity analysts said might be heavily hit by the earthquake disaster helped support the london market by showing only narrow losses in early trading <eos>
Predicted: the . s . s trading sell said which <unk> traders market said the be able in the the earthquake
Expected: this will require us to define and <unk> what is necessary or appropriate care <eos>
Predicted: <unk> is be the $ <unk> <unk> <unk> <unk> is the to <unk> and or
As you can see Transformer fails to grasp grammar compared to RNN. Is there anything wrong with my understanding?
EDIT
This is one example that caught my eye
Expected: also the big board met with angry stock specialists <eos>
Predicted: also met specialists board met the stock big with after
Most of the words predicted have is from the expected but in a different order. I have read that transformers are permutation invariant which is the reason why we include positional encoding with the word embedding.
