I am trying to use a Keras LSTM neural network for character level language modelling. As the input, I give it the last 50 characters and it has to output the next one. It has 3 layers of 400 neurons each. For the training data, I am using 'War of The Worlds' by H.G. Wells which adds up to 269639 training samples and 67410 validation samples.

After 7 epochs the validation accuracy has reached 35.1% and the validation loss has reached 2.31. However, after being fed the first sentence of war of the worlds to start it outputs:

the the the the the the the the the the the the the the the the...

I'm not sure where I'm going wrong; I don't want it to overfit and output passages straight from the training data but I also don't want it to just output 'the' repeatedly. I'm really at a loss as to what I should do to improve it.

Any help would be greatly appreciated. Thanks!


1 Answer 1


You can follow the below steps :

  1. LSTMs are slower in terms of convergence. They take much time to train amd thereby give better results. Try training the network for a more number of epochs like 50 or 65.
  2. Use smaller batch size. Try using a RMSProp optimizer. It takes a long time to converge but gives excellent results in case of recurrent nets.
  3. Also, try word level predictions. Means, train your LSTM in such a way that it can predict words and not characters.
  4. Use dropout layers in between two LSTM layers. Use a rate like 0.2 or 0.35.

As a working example, you can refer this project which generates poem lines using LSTM in tensorflow.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .