When using ChatGPT i have noticed it is fast on some questions and slow on others. Why is that?
Based on my understanding of the transformers architecture, the user prompt is tokenized and the vector of token ids is fed as input to a strictly feed-forward neural network and the output of the model becomes part of the input for the next token that the model has to generate (this gives the model its auto-regressive nature).
Since there are no conditional loops in the NN, I would expect the time to generate the first token should depend on the length of user prompt. It shouldn't matter whether the question being asked is easy or tough. But I have clearly observed that when asked an easy question ChatGPT responds very fast and when asked a tough question it takes time to respond. Why is that? Is it due to caching of previous responses?