As far as I understand, a Transformer has a specific input sequence length that depends on its architecture. So a model like
gpt-4 has a sequence length of 8192 tokens. As such, I am interested what happens when the input prompt is shorter than that. This question and answers suggest that the input is simply padded out to the full input sequence length. But that doesn't really seem plausible to me. That would mean that the cost of processing a 100 token prompt would be the same as the cost of processing a 1000 token prompt. And it clearly isn't, as they charge a different price for them - they charge for token count, not API request count.
I am interested to know how variable length input sequences are handled for experimenting with trying to implement my own language model. I have two possible hypothesis. One could be that they actually combine multiple prompts into a single large prompt and tell the model to complete them all. But I don't think that it is the case as the risk of mixing the prompts from different users and the data they have would probably drastically degrade the quality of the output. I also thought that maybe they have different model sizes - like
gpt4-3k, ... - but that seems like a large overhead, as each of these would need to be trained separately.