1
$\begingroup$

I am trying to use a Keras LSTM neural network for character level language modelling. As the input, I give it the last 50 characters and it has to output the next one. It has 3 layers of 400 neurons each. For the training data, I am using 'War of The Worlds' by H.G. Wells which adds up to 269639 training samples and 67410 validation samples.

After 7 epochs the validation accuracy has reached 35.1% and the validation loss has reached 2.31. However, after being fed the first sentence of war of the worlds to start it outputs:

the the the the the the the the the the the the the the the the...

I'm not sure where I'm going wrong; I don't want it to overfit and output passages straight from the training data but I also don't want it to just output 'the' repeatedly. I'm really at a loss as to what I should do to improve it.

Any help would be greatly appreciated. Thanks!

$\endgroup$

1 Answer 1

1
$\begingroup$

You can follow the below steps :

  1. LSTMs are slower in terms of convergence. They take much time to train amd thereby give better results. Try training the network for a more number of epochs like 50 or 65.
  2. Use smaller batch size. Try using a RMSProp optimizer. It takes a long time to converge but gives excellent results in case of recurrent nets.
  3. Also, try word level predictions. Means, train your LSTM in such a way that it can predict words and not characters.
  4. Use dropout layers in between two LSTM layers. Use a rate like 0.2 or 0.35.

As a working example, you can refer this project which generates poem lines using LSTM in tensorflow.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .