4
$\begingroup$

I am wondering why there has not been more usage of GANs for NLP. I know there has been research on the subject (The Google Scholar page for the subject is here).

Are there any specific reasons why GANs do not work for NLP specifically VQGAN + CLIP variants? I do not understand why most text generated by AI is done through predicting the next letter or word in a sequence with RNNs when GANs have had so much success generating deep fakes and the such instead of say, predicting the next pixel.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

A couple of reasons:

  1. Transformers are amazing at text generation already (e.g. GPT-3 which almost passes the Turing-Test)
  2. The original GAN requires a continuous data representation (e.g. images) instead of a discrete one (e.g. text), so that slight error signals can be used for learning.
  3. Empirically speaking, GANs don't seem to work that well on non-image data. I recently applied them to regular tabular data but found auto-encoders much more useful.
Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Please, remove the very arguable sentence "which essentially passes the Turing-Test". Alternatively, can you provide some real evidence for it? If it generates some meaningful text it doesn't mean passed the Turing test. I'm sure it doesn't pass the imitation game. $\endgroup$
    – nbro
    Mar 25, 2022 at 10:36
  • $\begingroup$ I edited it to include a reference, and say 'almost passes' since it still isn't great at answering nonsense questions. $\endgroup$
    – profPlum
    Mar 28, 2022 at 18:17
  • $\begingroup$ I should mention however that when GPT-3 is primed with examples (via meta-learning) of calling out nonsense it does learn the skill immediately: arr.am/2020/07/25/gpt-3-uncertainty-prompts $\endgroup$
    – profPlum
    Mar 28, 2022 at 18:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .