This question was flawed. See my answer with help from commenters.
This question has been asked in other forums   but I'm not sure I understand the claims, which are (EDIT: the following are based on my faulty assumption that pad tokens are added up to the maximum context window):
- Each forward pass takes less resources when more of the context window is padding.
- Forward passes are run on the input tokens.
- Forward passes with fewer non-pad input tokens are smaller tensor operations.
Hypothesis 1 seems the most plausible to me from a performance engineering standpoint (sparse math, etc). Does it fall out naturally from just writing basic JAX code or would it require manual optimization (if so, what tricks can be used?)? There does seem to be some research on this.
Hypothesis 2 and 3 seem wrong based on my surface-level understanding of the Transformer architecture.
I've tried a few open-source LLMs locally and neither on those nor ChatGPT have I noticed any difference in latency based on how much text was in the context window. But I haven't done actual rigorous benchmarking yet.
The reason this is relevant is due to document lookup-based applications. Looking at the OpenAI cookbook for Q&A using embeddings, I can see that:
- most of the token usage seems to come from the long Wikipedia article pasted into the context
- the price per query is quite prohibitive to frequent usage:
For gpt-3.5-turbo using ~1,000 tokens per query, it costs ~0.002 per query, or ~500 queries per dollar (as of Apr 2023) For gpt-4, again assuming ~1,000 tokens per query, it costs ~0.03 per query, or ~30 queries per dollar (as of Apr 2023)
My prior intuition would have been that optimal usage of LLMs would be to keep the context filled with inexpensive text (e.g. from NN search and/or cheaper LMs) and to have the LLM generate terse responses. But the input token cost model changes the strategy, as it means that users need to be sparing about the size and quantity of documents that they paste into the context window.