# Issue at training simple RNN for word generation

After completing Coursera course from Andrew Ng, I wanted to implement again simple RNN for generating dinosaurs name based on a text file containing around 800 dinosaurs name. This is done with Numpy in coursera, here is a link to a Jupyter notebook (not my repo) to get strategy and full objective: Here

I started similar implementation but in Pytorch, here is the model:

class RNN(nn.Module):
def __init__(self,input_size):
super(RNN, self).__init__()
print("oo")
self.hiddenWx1 = nn.Linear(input_size, 100)
self.hiddenWx2 = nn.Linear(100, input_size)
self.z1 = nn.Linear(input_size,100)
self.z2 = nn.Linear(100,input_size)
self.tanh = nn.Tanh()
self.softmax = torch.nn.Softmax(dim=1)

def forward(self, input, hidden):
layer = self.hiddenWx1(input)
layer = self.hiddenWx2(layer)
a_next = self.tanh(layer)
z = self.z1(a_next)
z = self.z2(z)
y_next = self.softmax(z)
return y_next,a_next


Here is the main algorithm of training:

for word in examples:  # for every dinosaurus name
hidden= torch.zeros(1, len(ix_to_char)) #initialise hidden to null, ix_to_char is below
word_vector = word_tensor(word) # convert each letter of  the current name in one-hot tensors
output = torch.zeros(1, len(ix_to_char)) #first input is null
loss = 0
counter = 0
true = torch.LongTensor(len(word)) #will contains the index of each letter.If word is "badu" => [2,1,4,22,0]

measured = torch.zeros(len(word)) # will contains the vectors returned by the model for each letter (softmax output)

for t in range(len(word_vector)): # for each letter of current word
true[counter] = char_to_ix[word[counter]] # char_to_ix return the index of letter in dictionary

output, hidden = model(output, hidden)

if (counter ==0):
measured = output
else: #measures is a tensor containing tensors of probability distribution
measured = torch.cat((measured,output),dim=0)
counter+=1

loss = nn.CrossEntropyLoss()(measured, true) #
loss.backward()
optimizer.step()


The letter dictionary (ix_to_char) is as follow:

{0: '\n', 1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z'}

Every 2000 epochs, I sample some new words with this function using torch multimonial to select a letter based on the softmax probability returned by the model:

def sampling(model):
idx = -1
counter = 0
newline_character = char_to_ix['\n']

x = torch.zeros(1,len(ix_to_char))
hidden = torch.zeros(1, len(ix_to_char))
generated_word=""

while (idx != newline_character and counter != 35):
x,hidden = model(x, hidden)
#print(x)
counter+=1
idx = torch.multinomial(x,1)
#print(idx.item())
generated_word+=ix_to_char[idx.item()]
if counter ==35:
generated_word+='\n'
print(generated_word)


Here are the results of the first display:

epoch:1, loss:3.256033420562744
aaasaaauasaaasasauaaaaapsaaaasaaaaa

aaaaaaaaaaaaasaaaoaaaaaauaaaaaaaaaa

taaaauasaasaaaaasaaasaauaaaaaaaausa

uaasaaaaauaaaasasssaauaaaaasaaaaaaa

auaaaaaaaassasaaauaaaaaaaaasasaaaas

epoch:2, loss:3.199960231781006
aaasaaassussssusssussssssssssusssss

aasaaassssssssssssasusssissssssssss

sasaaassssuosasssssssssssssssssssss

aasassasassusssssssssussssssssssuss

oasaasassssssussssssssussssssssssss

epoch:3, loss:3.263746500015259
aaaaaaasaaaasaaaaasaaaasaaaaaaaaaaa

aaaaaaasaaaaaaaaaaaaaaaaaaaaaaaaaaa

aaaaaaaaaaaaaaaaaaaaaauaaaaaaaaaaas

aaaaaaaasaaaasraaaaaaaaaaaaaaaaaaaa

aaaaaaaaaaaaauusaaaaauaaaaaaaaaaaaa


It doesn't work but I have no idea how to fix the issue. With no training at all, the sampling function seems to work as the returned words seem complete random:

hbtpsbykkxvlah

ttiwlzxdxabzmbdvsapsnwwpaoiasotalft


My post may be a bit long, but so far I have no idea what is the issue of my program.

• Can you add a link to the dataset. I'll check your model when I get a chance later but I'll need the dataset too. – respectful Sep 5 at 20:36
• Here is a link to the dataset (not my repo): github.com/carlb15/Dinosaur_Island_Character_Lvl_Language_Model/… thanks! – john7002 Sep 6 at 8:05
• I'm looking into it now. – respectful Sep 6 at 20:40
• thanks, just check again and seems that 'output' variable initialized as null vector get a its value updated after calling model the first time, and then this value is never updated.. – john7002 Sep 6 at 21:17
• That would explain all the a's – respectful Sep 6 at 21:19

Your forward function is not using the previous hidden state.

observe: you pass hidden but never use it.

def forward(self, input, hidden):
layer = self.hiddenWx1(input)
layer = self.hiddenWx2(layer)
a_next = self.tanh(layer)
z = self.z1(a_next)
z = self.z2(z)
y_next = self.softmax(z)
return y_next,a_next

• yes correct... sorry for the bad mistake, will modify it. – john7002 Sep 6 at 21:40
• it happens to the best of us. I hope this actually solves the issue. – respectful Sep 6 at 21:41