2
$\begingroup$

After completing Coursera course from Andrew Ng, I wanted to implement again simple RNN for generating dinosaurs name based on a text file containing around 800 dinosaurs name. This is done with Numpy in coursera, here is a link to a Jupyter notebook (not my repo) to get strategy and full objective: Here

I started similar implementation but in Pytorch, here is the model:

class RNN(nn.Module):
    def __init__(self,input_size):
        super(RNN, self).__init__()
        print("oo")
        self.hiddenWx1 = nn.Linear(input_size, 100) 
        self.hiddenWx2 = nn.Linear(100, input_size)
        self.z1 = nn.Linear(input_size,100)
        self.z2 = nn.Linear(100,input_size)
        self.tanh = nn.Tanh()
        self.softmax = torch.nn.Softmax(dim=1)

    def forward(self, input, hidden):
        layer = self.hiddenWx1(input)
        layer = self.hiddenWx2(layer)
        a_next = self.tanh(layer)
        z = self.z1(a_next)
        z = self.z2(z)
        y_next = self.softmax(z)
        return y_next,a_next

Here is the main algorithm of training:

for word in examples:  # for every dinosaurus name
                model.zero_grad()
                hidden= torch.zeros(1, len(ix_to_char)) #initialise hidden to null, ix_to_char is below
                word_vector = word_tensor(word) # convert each letter of  the current name in one-hot tensors
                output = torch.zeros(1, len(ix_to_char)) #first input is null
                loss = 0
                counter = 0
                true = torch.LongTensor(len(word)) #will contains the index of each letter.If word is "badu" => [2,1,4,22,0]

                measured = torch.zeros(len(word)) # will contains the vectors returned by the model for each letter (softmax output) 


                for t in range(len(word_vector)): # for each letter of current word
                    true[counter] = char_to_ix[word[counter]] # char_to_ix return the index of letter in dictionary

                    output, hidden = model(output, hidden)

                    if (counter ==0):
                        measured = output
                    else: #measures is a tensor containing tensors of probability distribution
                        measured = torch.cat((measured,output),dim=0)
                    counter+=1

                loss = nn.CrossEntropyLoss()(measured, true) #
                loss.backward()
                optimizer.step()

The letter dictionary (ix_to_char) is as follow:

{0: '\n', 1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z'}

Every 2000 epochs, I sample some new words with this function using torch multimonial to select a letter based on the softmax probability returned by the model:

def sampling(model):
    idx = -1 
    counter = 0
    newline_character = char_to_ix['\n']

    x = torch.zeros(1,len(ix_to_char))
    hidden = torch.zeros(1, len(ix_to_char))
    generated_word=""


    while (idx != newline_character and counter != 35):
        x,hidden = model(x, hidden)
        #print(x)
        counter+=1
        idx = torch.multinomial(x,1)
        #print(idx.item())
        generated_word+=ix_to_char[idx.item()]
    if counter ==35:
        generated_word+='\n'
    print(generated_word)

Here are the results of the first display:

epoch:1, loss:3.256033420562744
aaasaaauasaaasasauaaaaapsaaaasaaaaa

aaaaaaaaaaaaasaaaoaaaaaauaaaaaaaaaa

taaaauasaasaaaaasaaasaauaaaaaaaausa

uaasaaaaauaaaasasssaauaaaaasaaaaaaa

auaaaaaaaassasaaauaaaaaaaaasasaaaas

epoch:2, loss:3.199960231781006
aaasaaassussssusssussssssssssusssss

aasaaassssssssssssasusssissssssssss

sasaaassssuosasssssssssssssssssssss

aasassasassusssssssssussssssssssuss

oasaasassssssussssssssussssssssssss

epoch:3, loss:3.263746500015259
aaaaaaasaaaasaaaaasaaaasaaaaaaaaaaa

aaaaaaasaaaaaaaaaaaaaaaaaaaaaaaaaaa

aaaaaaaaaaaaaaaaaaaaaauaaaaaaaaaaas

aaaaaaaasaaaasraaaaaaaaaaaaaaaaaaaa

aaaaaaaaaaaaauusaaaaauaaaaaaaaaaaaa

It doesn't work but I have no idea how to fix the issue. With no training at all, the sampling function seems to work as the returned words seem complete random:

hbtpsbykkxvlah

ttiwlzxdxabzmbdvsapsnwwpaoiasotalft

My post may be a bit long, but so far I have no idea what is the issue of my program.

Manny thanks for your help.

$\endgroup$
  • $\begingroup$ Can you add a link to the dataset. I'll check your model when I get a chance later but I'll need the dataset too. $\endgroup$ – respectful Sep 5 at 20:36
  • $\begingroup$ Here is a link to the dataset (not my repo): github.com/carlb15/Dinosaur_Island_Character_Lvl_Language_Model/… thanks! $\endgroup$ – john7002 Sep 6 at 8:05
  • $\begingroup$ I'm looking into it now. $\endgroup$ – respectful Sep 6 at 20:40
  • $\begingroup$ thanks, just check again and seems that 'output' variable initialized as null vector get a its value updated after calling model the first time, and then this value is never updated.. $\endgroup$ – john7002 Sep 6 at 21:17
  • $\begingroup$ That would explain all the a's $\endgroup$ – respectful Sep 6 at 21:19
0
$\begingroup$

Your forward function is not using the previous hidden state.

observe: you pass hidden but never use it.

def forward(self, input, hidden):
    layer = self.hiddenWx1(input)
    layer = self.hiddenWx2(layer)
    a_next = self.tanh(layer)
    z = self.z1(a_next)
    z = self.z2(z)
    y_next = self.softmax(z)
    return y_next,a_next
$\endgroup$
  • $\begingroup$ yes correct... sorry for the bad mistake, will modify it. $\endgroup$ – john7002 Sep 6 at 21:40
  • $\begingroup$ it happens to the best of us. I hope this actually solves the issue. $\endgroup$ – respectful Sep 6 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.