There is a recent trend in people using LSTMs to write novels. I haven’t attempted this myself. From what I’m hearing, they can tell a story, but it seems they lose the context of the story rather quickly. After which they begin constructing new, but not necessarily related constructs.

Can they construct a plot in the long term?

  • 1
    $\begingroup$ An LSTM has trouble remembering over very long time frames. It also, as nbro said, can't really learn common-sense knowledge. It means it can't truly determine what information is important to remember (like a character dying for example) consistently. It very easily loses context when you let it run for extended periods of time (especially if it's longer then the input it was trained on), so it can't realistically write a complete story coherently. Even if you just wanted a short story (~200 words), it is very likely it would lose some important contextual knowledge. $\endgroup$
    – Recessive
    Nov 8 '19 at 23:50

The long-short term memory (LSTM) is a type of recurrent neural network, which is only suited for sequence modelling, that is, to keep track of statistical dependencies between elements of a sequence.

The LSTM prediction capabilities are limited to the training data that is used to train it, the inductive bias (in the case of LSTMs, the inductive bias particularly refers to the fact that elements of a sequence are dependent on each other) and the available computation resources. However, storytelling often assumes the existence of a common-sense knowledge between the storyteller and the listener (but LSTM completely ignores this) and requires the true understanding of language, which is believed to be an AI-complete problem (in simple words, it is a very complex task, which probably cannot be exactly solved with statistical models).

Furthermore, even though LSTM partially addresses the vanishing gradient problem (and they were specifically created to partially solve this issue), they can still suffer from the exploding gradient problem. Moreover, although LSTMs and, in general, neural networks are universal function approximators, in practice, functions may not be continuous, which is an assumption made in the universal approximation theorems.

  • $\begingroup$ Just a minor comment: it is probably worth noting that exploding gradients can be fixed easily with gradient clipping in most cases. $\endgroup$ Jan 29 '20 at 13:05
  • $\begingroup$ @MathiasMüller Thanks for your feedback and thanks for contributing to the site! I've actually not worked much with LSTMs (even though I am quite familiar with the theory), so your feedback is definitely valuable (especially, if you had hands-on experience with LSTMs). Clearly, I am aware of gradient clipping, but I am not sure how much it can help in this specific case, because I am not sure I've ever used it (in this case). $\endgroup$
    – nbro
    Jan 29 '20 at 13:09
  • 1
    $\begingroup$ Do you mean, this specific task of telling a story in particular? I don't know, I don't work on story generation. Some technical reading about exploding gradients in RNNs that you might not be aware of. The authors argue that LSTMs fix vanishing gradients to some extent, but they (= LSTMs) do not fix exploding gradients. The authors propose clipping the gradient norm. $\endgroup$ Jan 29 '20 at 13:46
  • $\begingroup$ @MathiasMüller No, I didn't mean storytelling in particular. I had already come across this paper, but I think I've never fully read it. Thanks for pointing this out. If I have some time, I will have a deeper look at it ;) $\endgroup$
    – nbro
    Jan 29 '20 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.