I am trying to use a Keras LSTM neural network for character level language modelling. As the input, I give it the last 50 characters and it has to output the next one. It has 3 layers of 400 neurons each. For the training data, I am using 'War of The Worlds' by H.G. Wells which adds up to 269639 training samples and 67410 validation samples.
After 7 epochs the validation accuracy has reached 35.1% and the validation loss has reached 2.31. However, after being fed the first sentence of war of the worlds to start it outputs:
the the the the the the the the the the the the the the the the...
I'm not sure where I'm going wrong; I don't want it to overfit and output passages straight from the training data but I also don't want it to just output 'the' repeatedly. I'm really at a loss as to what I should do to improve it.
Any help would be greatly appreciated. Thanks!