How to optimize transformer inference for prompts shorter than the maximum sequence length?

Question

As far as I understand, a Transformer has a specific input sequence length that depends on its architecture. So a model like gpt-4 has a sequence length of 8192 tokens. As such, I am interested what happens when the input prompt is shorter than that. This question and answers suggest that the input is simply padded out to the full input sequence length. But that doesn't really seem plausible to me. That would mean that the cost of processing a 100 token prompt would be the same as the cost of processing a 1000 token prompt. And it clearly isn't, as they charge a different price for them - they charge for token count, not API request count.

I am interested to know how variable length input sequences are handled for experimenting with trying to implement my own language model. I have two possible hypothesis. One could be that they actually combine multiple prompts into a single large prompt and tell the model to complete them all. But I don't think that it is the case as the risk of mixing the prompts from different users and the data they have would probably drastically degrade the quality of the output. I also thought that maybe they have different model sizes - like gpt4-1k, gpt4-2k, gpt4-3k, ... - but that seems like a large overhead, as each of these would need to be trained separately.

Answer 1

As @N. Kiefer wrote, a trained transformer can handle input sequences of a length smaller than it maximum sequence length. I thought about this and I realized how this is a direct consequence of transformers' design.

Input embeddings are augmented with positional encoding. This works "out of the box" as the positional encoding is a function of the token placement within the input sequence [0, max_seq_len]. So the positional encoding happens as usual.

Then, they go through the encoder stack. In a single encoder, they are first converted to Query, Key and Value matrices. This conversion uses predetermined per-head matrices. Then Z matrices are computed per each token per each head. This also works as usual, as the sizes of Q, V, and K match. Combining multi-headed attention also works, as this is, again, a per token process. Then, the residual original encoding is added on top and the values proceed to the linear layer, where each token is processed individually. So, apparently, the encoder works fine with different input lengths.

Encoder-decoder cross-attention also works. That's because the Q matrix taken from decoder's masked self-attention can be multiplied with any number of keys from the encoder. And the key and value matrix sizes match, as they are both coming from the encoder.

Answer 2

you only pad sequences when you combine sequences of different lengths. This way you get a tensor of same sized sequences. when you only want to process one sequence, Transformers are capable of handling different sizes.