0
$\begingroup$

When using ChatGPT i have noticed it is fast on some questions and slow on others. Why is that?

Based on my understanding of the transformers architecture, the user prompt is tokenized and the vector of token ids is fed as input to a strictly feed-forward neural network and the output of the model becomes part of the input for the next token that the model has to generate (this gives the model its auto-regressive nature).

Since there are no conditional loops in the NN, I would expect the time to generate the first token should depend on the length of user prompt. It shouldn't matter whether the question being asked is easy or tough. But I have clearly observed that when asked an easy question ChatGPT responds very fast and when asked a tough question it takes time to respond. Why is that? Is it due to caching of previous responses?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

That is a very good question. On a high level, it might be the Causal reasoning used by OpenAI and designed by their Legal and Design teams.

WARNING! Trigger words ahead - Completely for illustration purposes.

As you can see, in the first completion produced by Da-vinci-003, it produced the content based on my input alone. But, as we know their chat models are fine-tuned for human level conversations, and with time have also been explicitly trained to do and not-do certain things, it refused to answer my query. What I believe is that they have multiple causal layers, which get triggered with certain queries.

Now with that argument, this causal layer, when recieves a sophisticated query, it is instructed to generate a certain kind of response. Maybe it is instructed to first break it down into simpler points, structure them in a logical order before generating the "to be returned" response to the user.

On the technical side, you are correct, during the inference the time complexity should practically be the same (given two inputs of the same number of tokens).

  1. https://i.stack.imgur.com/5UA8n.png [davinci-003]

  2. https://i.stack.imgur.com/1T8il.png [gpt-3.5-turbo]

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ This site not appropriate for speculations. This site originally fell under the science category, so it should be about facts. So, I'd discourage you from providing answers if you only have a guess. If you have a guess, you can leave a comment, once you get the privilege to comment under posts. $\endgroup$
    – nbro
    Jun 30 at 9:23
  • $\begingroup$ @nbro makes sense. I will keep that in mind. $\endgroup$
    – Chinmay
    Jun 30 at 16:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .