I am diving in data-to-text generation for long articles (> 1000 words). After creating a template and fill it with data I am currently going down on paragraph level and adding different paragraphs, which are randomly selected and put together. I also added on a word level different outputs for date, time and number formats.

The challenge I see is, that when creating large amounts of such generated texts they become boring to read as the uniqueness for the reader goes down.

Furthermore, I also think it's easy to detect that such texts have been autogenerated. However, I still have to validate this hypotheses.

I was wondering if there is an even better method to bring in variability in such a text?

Can you suggest any methods, papers, resources or share your experience within this field.

I highly appreciate your replies!


The state of the art in text generation is the GPT model. GPT-3, which was just released in summer of 2020, has been used to generate many very impressive articles, and is widely considered the best text generation model. This article and this one should give you an example of how powerful it is at text generation.

GPT is a transformer based architecture, somewhat like BERT. The main difference is it only takes into account left context, which is why it is so well suited for text generation.

GPT-3 is still very new and is not available for free. However, GPT-2, the previous release of the model, is available for free. While obviously not as advanced as GPT-3, it is still quite impressive in its own right, and for someone trying to generate text, it is the clear choice.

Here is a link to a tutorial explaining the basics to get you started with GPT-2.

If you are interested in diving into the relevant research papers:

Here are the open AI papers on GPT 1-3

  1. Improving Language Understanding by Generative Pre-Training
  2. Language Models are Unsupervised Multitask Learners
  3. Language Models are Few-Shot Learners

Additionally, if you have never seen transformers before, take a look at:

  1. Attention Is All You Need

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.