Are there any good alternatives to an LSTM language model for text generation?

Question

I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be:

1. Apply softmax function
2. Take a weighted random choice to determine the next word

This is working reasonably well for me, but it would be nice to play around with other options. Are there any good alternatives to this?

Answer 1

The current state of the art in natural language generation are all auto-regressive transformer models. Transformers no longer use recurrent neural networks such as LSTM, because the recurrences makes long dependencies messy to calculate. Instead, Transformers only keep the attention layers, and apply attention on all the existing text so far, which can be done in parallel so therefore very fast, while being able to attend to long dependencies (e.g. understanding that "it" refers to "John" from 3 sentences ago). They are also faster to train than LSTMs (on powerful GPUs at least). The downside is more memory requirement, and you need large models and large datasets (LSTMs work better for small models and small datasets). Here is some background info on how they work.

Auto-regressive transformer models only use the decoder for text generation, and removes the encoder. Given an input, they predict the next word.

The most well-known one is GPT (GPT-3 has 175B parameters; GPT-2 has 1.5B parameters, and GPT-1 has 175M parameters) GPT is developed by OpenAI and is a commercial, paid-software if you want to use their official model, but I'm sure with a little digging you can find community-trained models that will perform slightly worse but is at least free to use. GPT is basically a vanilla transformer, but trained on a huge, huge dataset with a huge, huge model to achieve state-of-the-art performance.

Other auto-regressive transformer models include:

• CTRL by Salesforce, which uses the novel idea of control codes to guide the style of generation (e.g. to generate Wikipedia article style text or book review style text).
• XLNet by Google AI Brain team, which handles longer sequences more accuractely than the others because it re-introduces reccurrence back in the transformer model, allowing it to remember past sequences. Otherwise, vanilla transformers cannot handle dependencies that crosses sequences (note: a sequence is limited by the max length you can feed into the model, bottle-necked by your memory requirement, and can contain many sentences or paragraphs).
• Reformer by Google Research, which is a more efficient transformer that significantly reduces the memory requirement while also being faster to compute on long sequences.

If your goal is just to generate English or another commonly researched language, you can use an existing pre-trained language model and avoid doing any training yourself. This saves a lot of time, and there should at least be free community-trained models readily available. Otherwise, for obscure tasks, you'd have to train one yourself, and these state of the art models will take immense resources.