There is a recent trend in people using LSTMs to write novels. I haven’t attempted this myself. From what I’m hearing, they can tell a story, but it seems they lose the context of the story rather quickly. After which they begin constructing new, but not necessarily related constructs.

Can they construct a plot in the long term?

  • 1
    $\begingroup$ An LSTM has trouble remembering over very long time frames. It also, as nbro said, can't really learn common-sense knowledge. It means it can't truly determine what information is important to remember (like a character dying for example) consistently. It very easily loses context when you let it run for extended periods of time (especially if it's longer then the input it was trained on), so it can't realistically write a complete story coherently. Even if you just wanted a short story (~200 words), it is very likely it would lose some important contextual knowledge. $\endgroup$ – Recessive Nov 8 '19 at 23:50

The long-short term memory (LSTM) is a type of recurrent neural network, which is only suited for sequence modelling, that is, to keep track of statistical dependencies between elements of a sequence.

The LSTM prediction capabilities are limited to the training data that is used to train it, the inductive bias (in the case of LSTMs, the inductive bias particularly refers to the fact that elements of a sequence are dependent on each other) and the available computation resources. However, storytelling often assumes the existence of a common-sense knowledge between the storyteller and the listener (but LSTM completely ignores this) and requires the true understanding of language, which is believed to be an AI-complete problem (in simple words, it is a very complex task, which probably cannot be exactly solved with statistical models).

Furthermore, even though LSTM partially addresses the vanishing gradient problem (and they were specifically created to partially solve this issue), they can still suffer from the exploding gradient problem. Moreover, although LSTMs and, in general, neural networks are universal function approximators, in practice, functions may not be continuous, which is an assumption made in the universal approximation theorems.


The bottleneck is the underlying heuristics model, namely the problem which story make sense and which not. A possible attempt in automatic story telling is to produce first a sequence of images which show the plot visual. In the next step, the images are converted into natural language. The means, it's told to the audience what they can see on the picture.

The abstract problem is, which kind of image sequence make sense. For example, before a human can enter a house, he has to go to the house. Formalizing such constraints can be realized with a graph. There are some pictures available which are equal to the nodes, and the edges between the nodes are describing possible pathways through the images. The same node can become part of different plots.

Like i mentioned before, the problem isn't located in the LSTM network itself but in the environment. Any neural network is able to learn the path between nodes. A normal three layer backpropagation neural network is more than capable for the job. The more serious problem is how to formalize the requirement of telling a story, into a neural network representation. Which is equal to 0 and 1 binary items.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.