5
$\begingroup$

When you give ChatGPT a prompt, does it

A) generate the entire response instantly (at least on the server), but then "buffers" it slowly to the end user (for whatever reason)

or

B) it's actually generating words (and sometimes entire sentences at once) one by one, and using (server) computational power all that while?

$\endgroup$
1
  • $\begingroup$ This has already been asked here. $\endgroup$
    – nbro
    Feb 21, 2023 at 19:14

1 Answer 1

4
$\begingroup$

The model infers in an autoregressive way. That is, it needs the previous output to produce the next one, so it's slow because it computes. More details.

It also depends on the language. The model was trained primarily on the English corpus and typically generates one or two tokens for each word. But for other languages, it might not be as efficient and can generate a token for each symbol

$\endgroup$
1
  • 1
    $\begingroup$ Also you might notice that sometimes it will type out a response and then replace the whole text with a standard response once some other system finds out that the GPT output is violating the their policy in some way. GPT is slow, and the policy checking algorithm is run on intermediate output from GPT. $\endgroup$ May 3, 2023 at 9:45

Not the answer you're looking for? Browse other questions tagged .