My understanding of the language model is that the output of the model is a tensor. So the whole output should be computed all together. But why ChatGPT like models can output one token at a time like a stream? And the time difference between the output of first and last token is significant.

  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Commented Apr 1, 2023 at 7:14
  • $\begingroup$ The output of a GPT model is a probability for each possible next token. $\endgroup$ Commented Apr 1, 2023 at 8:09
  • $\begingroup$ See ai.stackexchange.com/questions/38923/… $\endgroup$
    – Rexcirus
    Commented Apr 1, 2023 at 23:20


Browse other questions tagged .