16
$\begingroup$

When ChatGPT is generating an answer to my question, it generates it word by word.
So I actually have to wait until I get the final answer.

Is this just for show?
Or is it really real-time generating the answer word by word not knowing yet what the next word will be?

Why does it not give the complete answer text all at once?

Improve this question
$\endgroup$
7
  • 7
    $\begingroup$ It is just for show. $\endgroup$
    – Dr. Snoopy
    Jan 27, 2023 at 15:54
  • 2
    $\begingroup$ It also does in fact generate the output 1 token at a time, not "knowing" what the next word will be, because each token added to the sequence changes the probablities for further tokens, and this process is necessarily sequential. However, it is unlikely to be doing a whole round trip between the model running on a GPU and the web interface for each token, that would be very inefficient. $\endgroup$
    – Neil Slater
    Jan 27, 2023 at 20:23
  • 1
    $\begingroup$ It can also be for inducing us to read it all. Should it dump it all at once, we would be more likely to skim over the answer or even quit. It helps engagement. $\endgroup$
    – Jaume Oliver Lafont
    Mar 1, 2023 at 10:00
  • $\begingroup$ @Dr.Snoopy No, it is not just for show. It literally does not know the later tokens when it generates a token, the output is real time. $\endgroup$
    – Volker Siegel
    Apr 13, 2023 at 19:49
  • 2
    $\begingroup$ @Dr.Snoopy If you use the API, you can specify stream=True if you to not want the server to wait until the whole response is ready, but return it as many small "events". A large completion can take quite a long time, so it makes a big difference wheter you get the first result after few seconds or a minute. On the api level, it would be somewhat absurd to have a built in delay on the server side. And the delay you see in a web interface behaves exactly like the API. github.com/openai/openai-cookbook/blob/main/examples/… $\endgroup$
    – Volker Siegel
    Apr 14, 2023 at 19:53

2 Answers 2

Reset to default
9
$\begingroup$

ChatGPT is a conversational-agent based on GPT3.5, which is a causal language model. Under the hood, GPT works by predicting the next token when provided with an input sequence of words. So yes, at each step a single word is generated taking into consideration all the previous words.

See for instance this Hugging Face tutorial.

To further explain: while outputting entire sequences of words is in principle possible, it would require a huge amount of data, since the probability of each sequence in the space of all sequences is extremely small. Instead, building a probability distribution over half a million of english words is feasible (in reality just a tiny fraction of those words is often used).

On top of that, there may be some scenic effect, to simulate the AI "typing" the answer.

Improve this answer
$\endgroup$
4
  • 5
    $\begingroup$ I think you may be right, given my knowledge of these models. However, ChatGPT could also generate all text (word-by-word or not) before showing it to the user, then show it to the user all at once. So, in a way, this is also for show. $\endgroup$
    – nbro
    Feb 1, 2023 at 9:57
  • $\begingroup$ Although the first three paragraphs of your answer are true, I'd be surprised if they were actually the direct cause of displaying the answer one word at a time. More likely, this is just for scenic effect, as per your last sentence, and perhaps also to slow the conversation down so that the server doesn't receive too many requests at all. $\endgroup$
    – Stef
    Mar 31, 2023 at 15:29
  • $\begingroup$ @nbro That would be just collecting output on the server side to return it as one document. The API actually does that, it is easier to use in programming when you do not care about the time and would otherwise need to collect the parts yourselves. You specify stream=True in the API for incremental results. $\endgroup$
    – Volker Siegel
    Apr 14, 2023 at 20:04
  • $\begingroup$ @VolkerSiegel Yes, I've worked with their quite unreliable (slow, often unavailable, etc.) API. $\endgroup$
    – nbro
    Apr 17, 2023 at 12:07
4
$\begingroup$

Why does ChatGPT not give the answer text all at once?

Because ChatGPT is autoregressive (=generates each new word by looking at previous words), as Rexcirus mentioned.

Is this just for show?

On https://beta.openai.com/playground, output words/tokens are displayed faster when using smaller models such as text-curie-001 than larger ones such as text-davinci-003. I.e., the inference time does seem to impact the display time.

https://twitter.com/ArtificialAva/status/1624411499375603715 compared the display speed of ChatGPT vs. ChatGPT Plus vs. ChatGPT Turbo Mode and showed that ChatGPT Turbo Mode is over twice faster to display the output, which further indicates that ChatGPT shows its response word by word due to its backend (computation time + autoregressive).

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Or maybe the chatbot is just intentionally slowing the conversation down by displaying the words slowly, to avoid the server drowning under the huge number of users. And then if you register for turbo mode then the server gives you higher priority, and displays the words faster, regardless of how fast they are actually generated by the algorithm. $\endgroup$
    – Stef
    Mar 31, 2023 at 15:26
  • 1
    $\begingroup$ @Stef No, it is all real time output from different models, it is more clear when you use the API directly. $\endgroup$
    – Volker Siegel
    Apr 13, 2023 at 19:52
  • $\begingroup$ @VolkerSiegel Definitely, we've used the API quite a lot for arxiv.org/pdf/2304.05613.pdf and some other projects. $\endgroup$
    – Franck Dernoncourt
    Apr 27, 2023 at 3:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .