How LLM keeps the context of a chat/thread

Question

How an LLM keeps the context (what has already been entered by the user) of a chat/thread?

For reference, in chat.openai.com, for each chat we create (or a Thread according to their API), the LLM remembers what we have already input to the model, when answering a new question.

I did some reading on the topic and found below possible ways:

1. change the weights accordingly: but this seems not-practical for LLM given their size (even changing weights of the last layer seems an over-kill)
2. output a context vector at each inference and re-use it for the next inference: this seems more likely. but I am not sure exactly how to do it.

It would be great if someone can help me with this.

Thanks.

Answer 1

The thread maintains a content window.

All LLMs have fixed content window size which is measured in tokens like 2048 tokens. So, When generating a Question it will consider only that many tokens as context.

As we do the conversation the model stores the token from the conversation inside the content window which includes user input and model response.

When the size of the conversation exceeds the content window size it will drop the past tokens.

So, with the next question or conversation sentence old content window also goes as context.

In terms of memory, it is stored as a session content in DB or similar.