As far as I understand, beam search is the most widely used algorithm for text generation in NLP. So I was wondering: does the human brain also use beam search for text generation? If not, then what?
$\begingroup$
$\endgroup$
5
-
2$\begingroup$ i dont think neuroscience is far enough to the point it can answer that.... beamsearch is an effective and memory constrained method of approximating the sequence that maximizes the joint probability $\endgroup$ – mshlis Jun 2 '19 at 19:56
-
$\begingroup$ I think text generation uses for more different approaches than just this one algorithm. There is a lot of NLP happening which does not involve neural networks/deep learning approaches. $\endgroup$ – Oliver Mason Jun 3 '19 at 8:29
-
$\begingroup$ @OliverMason for example ... ? $\endgroup$ – Pablo Messina Jun 5 '19 at 17:35
-
$\begingroup$ Any work in text generation in the past 60 years. $\endgroup$ – Oliver Mason Jun 6 '19 at 8:17
-
$\begingroup$ @OliverMason would you mind being more specific? For example, any state of the art method which does not use beam search? Some concrete references/links will be appreciated. $\endgroup$ – Pablo Messina Jun 6 '19 at 21:38
Add a comment
|
$\begingroup$
$\endgroup$
I thought the answer might be no.
In this 2020 ICLR paper: The Curious Case of Neural Text Degeneration, researchers found that beam search text is less surprising compared to human natural language. And they proposed a nucleus sampling method which generates more human like text.
