1
$\begingroup$

I'm trying to understand how context length is calculated in transformer models, especially in NLP tasks using open-source models like llama, which can be run locally. I'm confused about how this is defined and whether hardware specifications (like GPU/CPU memory) affect the context length.

From what I understand, the maximum context length might be limited by memory, but I don't often see hardware specifications mentioned when discussing context length limits. For models that can be run locally, how exactly does hardware influence this, and why isn't it typically part of the context length discussion when dealing with transformer models? for example when they say llama 3.1 context length is 128k, based on what requirements they say that ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Context length is fixed

128K refers to Llama 3.1's ability to take up to 128,000 tokens in a single forward pass. This limit is fixed during training.

Inference on local hardware

When running inference locally (assuming your question isn't about hardware for training), your hardware significantly influences the speed of the model's performance but doesn't affect the fixed context length. There are optimization techniques, such as offloading model layers between RAM and VRAM and using quantization to reduce floating-point precision (which may impact response quality).

But why "128"?

According to the to the paper:

We increase the supported context length in increments, pre-training until the model has successfully adapted to the increased context length. We assess successful adaptation by measuring whether (1) model performance on short-context evaluations has recovered completely and (2) the model perfectly solves "needle in a haystack" tasks up to that length. In Llama 3 405B pre-training, we increased context length gradually in six stages, starting from the original 8K context window and ending in the final 128K context window. This long-context pre-training stage was performed using approximately 800B training tokens.

The "needle in a haystack" test involves hiding a key fact within a large amount of information. The model is then asked a question that requires that specific fact to answer correctly. This tests the model's ability to recover that small fact in a such a large context, it would catch if it's answering solely on pre-existing knowledge within it's weights for instance (and not using the context effectively).

It appears that the "128k" was chosen based on the quality of the response instead of on hardware specifications.

P.S.: It's a bit odd to me that they tried "six stages" from 8K and 128K instead of five (8K, 16K, 32K, 64K, 128K).

Improve this answer
$\endgroup$
4
  • $\begingroup$ To increase the context length the model would need to be retrained. have I skipped a lecture on how a transformer works? $\endgroup$
    – Alberto
    Commented Sep 18 at 19:55
  • $\begingroup$ Hi @Alberto, I've removed the sentence because it might be indeed misleading, thank you for the comment. My understanding is that if you increase the context length ideally you have to retraining/fine tuning it, to better generalize with such bigger context length. I confess I never tested it myself, so I wonder how poorer it will be. $\endgroup$
    – talles
    Commented Sep 18 at 23:15
  • $\begingroup$ Thanks, so do you mean that in inference time if my hardware allows, i can put as many as tokens that i want and pass the 128k limit but it will perform poorly or it gives an error ? $\endgroup$
    – Parsa Foroozmand
    Commented Sep 19 at 8:51
  • $\begingroup$ @ParsaForoozmand the architecture has no limitation on the input length, however they are still ML models, thus trained on a distribution of data (in this case, the max-length of the sentences), thus even though it allows it, it might have very poor performances $\endgroup$
    – Alberto
    Commented Sep 19 at 13:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .