1
$\begingroup$

Is there a limitation in current large language models (LLMs) in terms of practical processing time or memory resources when it comes to digesting the context provided by users? What I mean regarding context is the data fed to LLMs post-training, like feeding a selection of books about a certain topic.

$\endgroup$

1 Answer 1

3
$\begingroup$

Most state of the art LLMs nowadays are based on an architecture called Transformer, which uses a technique called attention, which scales quadratically with the input size. Thus, for an input of 1 million tokens, you'll have to create a matrix of size $(10^6)^2 = 10^{12}$ numbers, which are usually float32, which are composed of 4 bytes each. In other words, a matrix that is 4 Terabytes in size.

Alternatives have been proposed. For example linear transformers, sparse attention, RMTs, and quantization, but they are not as effective as the full transformer model.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .