I am a new learner in NLP. I am interested in the sentence generating task. As far as I am concerned, one state-of-the-art method is the CharRNN, which uses RNN to generate a sequence of words.

However, BERT has come out several weeks ago and is very powerful. Therefore, I am wondering whether this task can also be done with the help of BERT? I am a new learner in this field, and thank you for any advice!

  • $\begingroup$ Can OpenAI GPT be used for this? I believe OpenAI GPT has different architecture and is used for text generation $\endgroup$ – user10557045 Mar 20 at 16:17


Recent research shows that BERT is an MRF language model. Thus you can begin with a sentence of [MASK] tokens and generate words one by one in arbitrary order (instead of the common left-to-right chain decomposition).

Here's the paper: BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model. The code is also open sourced.

Though BERT is capable of text generation, the best text generation model in my mind is OpenAI's GPT, especially recently published GPT-2.


this experiment by Stephen Mayhew suggests that BERT is lousy at sequential text generation:


although he had already eaten a large meal, he was still very hungry

As before, I masked “hungry” to see what BERT would predict. If it could predict it correctly without any right context, we might be in good shape for generation.

This failed. BERT predicted “much” as the last word. Maybe this is because BERT thinks the absence of a period means the sentence should continue. Maybe it’s just so used to complete sentences it gets confused. I’m not sure.

One might argue that we should continue predicting after “much”. Maybe it’s going to produce something meaningful. To that I would say: first, this was meant to be a dead giveaway, and any human would predict “hungry”. Second, I tried it, and it keeps predicting dumb stuff. After “much”, the next token is “,”.

So, at least using these trivial methods, BERT can’t generate text.


No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). Because of bi-directionality of BERT, BERT cannot be used as a language model. If it cannot be used as language model, I don't see how you can generate a sentence using BERT.

  • 1
    $\begingroup$ My answer is no longer correct. You may want to accept @soloice 's answer $\endgroup$ – Astariul Feb 18 at 23:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.