Based on nothing but my own experience trying to build a similar chatbot using text-davinci-003, I think they are using the model itself to summarize the conversation, then feeding that summary back into the prompt. I get good results when doing this - though obviously the team behind ChatGPT does a better job.
Here is an example using the discussion under the question above:
You are a user named "AI" in an online forum. Below is a conversation
about a question. Please add your contribution to the conversation.
Question: """
One of the innovations with OpenAI's ChatGPT is how
natural it is for users to interact with it.
What is the technical enabler for ChatGPT to maintain the context of
previous questions in its answers? For example, ChatGPT understands a
prompt of "tell me more" and expands on it's previous answer.
Does it use activations from previous questions? Is there a separate
input for the context? How does it work? """
Rainb: """
ChatGPT says:
"ChatGPT uses a type of natural language processing called
"transformer" architecture, which is a type of neural network that is
designed to process and generate language. One of the key features of
transformer architecture is that it allows the model to maintain a
"context" of previous inputs. This is achieved through the use of
self-attention mechanisms, which allow the model to weight different
parts of the input differently based on how relevant they are to the
current context. This allows the model to generate more nuanced and
context-aware responses." """
milez: """
It does, indeed, but the
transformer architecture alone does not explain this capability.
Maintaining context requires something else. In fact, I think being a
transformer is irrelevant for this purpose. """
Rexcirus: """
My best
guess is that the whole conversation is fed as input for the next
reply. But looking into openai labelling guidance, it looks that the
data collection is single Q&A only. """
Rexcirus: """
If my guess is
correct, when the conversation is very long ChatGPT should start
forgetting previous bits, due to the maximum sequence size. """
yters:
"""
The ability to maintain context is very mysterious. What is most
mysterious to me is that if I continuously converse with ChatGPT it
has no problem maintaining context. On the other hand, if I break in
the middle of a conversation, and then come back to it some time
later, it's as if ChatGPT has had a memory wipe and has a lot of
trouble regaining the thread of the conversation. Since ChatGPT is a
neural network where the output is generated based on the previous
2048 input tokens, the amount of time between responses should have no
bearing on the output generated. """
milez: """
Great observation
@yters. Could it be a bug: the interface correctly shows the previous
conversation, but the service fails to send this to the OpenAI APIs
after some time. """
AI: """
It is possible that the OpenAI API may
be designed to automatically discard the context after some time to
avoid the system being overburdened with large conversations. Another
possibility is that the system may be storing the context, but using a
different mechanism than the transformer architecture to access it.
"""
Now, I'll ask it to summarize the conversation so far:
You are a user named "AI" in an online forum. Below is a conversation
about a question. Please summarize the interaction so far.
[I then pasted in everything from the Question down in the previous prompt/response]
The conversation so far has focused on understanding the technical
enablers that allow ChatGPT to maintain context in its answers. Rainb
has suggested that this is done through transformer architecture and
self-attention mechanisms. Rexcirus has suggested that the whole
conversation may be fed as input for the next reply. Yters has then
posed the interesting question of why, when the conversation is broken
off and then resumed, ChatGPT has difficulty regaining the thread.
Milez has suggested that this could be a bug in the interface.
Finally, I delete most of the conversation except for the summary and last few responses and alter the prompt a little to alert it to the presence of the summary:
You are a user named "AI" in an online forum. Below is a summary of
the conversation so far about a question, and the last few responses
to that question. Please add your contribution to the conversation.
Summary: The conversation so far has focused on understanding the
technical enablers that allow ChatGPT to maintain context in its
answers. Rainb has suggested that this is done through transformer
architecture and self-attention mechanisms. Rexcirus has suggested
that the whole conversation may be fed as input for the next reply.
Yters has then posed the interesting question of why, when the
conversation is broken off and then resumed, ChatGPT has difficulty
regaining the thread. Milez has suggested that this could be a bug in
the interface.
yters: """
The ability to maintain context is very mysterious. What is
most mysterious to me is that if I continuously converse with ChatGPT
it has no problem maintaining context. On the other hand, if I break
in the middle of a conversation, and then come back to it some time
later, it's as if ChatGPT has had a memory wipe and has a lot of
trouble regaining the thread of the conversation. Since ChatGPT is a
neural network where the output is generated based on the previous
2048 input tokens, the amount of time between responses should have no
bearing on the output generated. """
milez: """
Great observation
@yters. Could it be a bug: the interface correctly shows the previous
conversation, but the service fails to send this to the OpenAI APIs
after some time. """
AI: That is an interesting point, @Milez. It could be that the
interface does not accurately display the previous conversation to the
API, or that there is a bug in the API that causes it to not be able
to accurately generate a response with the 2048 input tokens when the
conversation is broken off and then resumed. It would be interesting
to investigate further.
I've had some success with this method, though it does still lose detail. Improving the prompt to create a more detailed summary helps. It's amazing how much of a conversation is full of repetitive filler, and how concisely it can be summarized. Maybe this is some kind of lingual compression? :)