27

For newbies, NO. Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its bidirectional nature. For advanced researchers, YES. You can start with a sentence of all [MASK] tokens, and generate words one by one in arbitrary order (instead ...


5

this experiment by Stephen Mayhew suggests that BERT is lousy at sequential text generation: http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/ although he had already eaten a large meal, he was still very hungry As before, I masked “hungry” to see what BERT would predict. If it could predict it correctly without any right context, we might be in ...


3

The current state of the art in natural language generation are all auto-regressive transformer models. Transformers no longer use recurrent neural networks such as LSTM, because the recurrences makes long dependencies messy to calculate. Instead, Transformers only keep the attention layers, and apply attention on all the existing text so far, which can be ...


3

Take the sentence that was generated by your LSTM and feed it back into the LSTM as input. Then the LSTM will generate the next sentence. So the LSTM is using it's previous output as it's input. That's what makes it recursive. The intial word is just your base case. Also you should consider using GPT2 by open AI to do this. It's pretty impressive. https://...


2

No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). Because of bi-directionality of BERT, BERT cannot be used as a language model. If it cannot be used as language model, I don't see how you can generate a sentence using BERT.


2

As you know, an LSTM language model takes in the past word and tries to predict the new one and continue over a loop. A sentence is divided into tokens and depending on different method, the tokens are divided differently. Some model maybe character based models which simply uses each character as input and output. In this case you can treat punctuation as ...


1

The state of the art in text generation is the GPT model. GPT-3, which was just released in summer of 2020, has been used to generate many very impressive articles, and is widely considered the best text generation model. This article and this one should give you an example of how powerful it is at text generation. GPT is a transformer based architecture, ...


1

In short: It depends. Where will you run it? On Premises: You may want want to run in your own environment. IaaS: GPT models are often too big, so people might prefer to setup a different server for that, serving your API. PaaS: If it's more experimental, I recommend running it on Google Colab. SaaS: Or even use some external API, so you don't need to worry ...


1

What alternate options exist for this? No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). Because of bi-directionality of BERT, BERT cannot be used as a language model. If it cannot be used as language model, I don't see how you can generate a sentence using BERT.


Only top voted, non community-wiki answers of a minimum length are eligible