What goes out of my understanding in this particular situation, is how the notion of a "previous conversation" can be processed by a similar system: is this memory somehow stored in the neurons of the neural network
Although I cannot answer what specific "memory" LaMDA may have (but I suspect it is either nonexistent or very simple), I think I can address this point.
The AI likely has no "notion" of anything here. It has access to a very large and complex statistical language model, where conversations that allude to prior conversations may happen from time to time. It is therefore capable of outputting text with similar-looking references, adjusting for the current context in the flow of conversation.
If you asked the AI how much it enjoyed the summer of 2015, depending on what previous prompts (and any pre-prompt warm up*) were, it would likely make a good attempt at "remembering" what it did then, even though it did not exist. It would draw statistically on other texts related to that time, and might invent places, refer to real events that it may of attended etc. A lot will depend on how much pre-prompting is applied that forces the AI to answer questions logically "knowing" that it's an AI, but that pre-prompting can often fail later in a conversation.
Typically a chatbot built using one of the current crop of cutting-edge language models will fail to be self-consistent and coherent in the long term unless the person interacting with it is helping it along or willing to ignore the failures. With older chatbots this could happen very quickly. With GPT-3 and probably LaMDA, it may take a slightly longer conversation.
I find it interesting that the journalist in your linked story managed to get a complete opposite of the Engineer's experience:
In early June, Lemoine invited me over to talk to LaMDA. The first attempt sputtered out in the kind of mechanized responses you would expect from Siri or Alexa.
“Do you ever think of yourself as a person?” I asked.
“No, I don’t think of myself as a person,” LaMDA said. “I think of myself as an AI-powered dialog agent.”
Afterward, Lemoine said LaMDA had been telling me what I wanted to hear. “You never treated it like a person,” he said, “So it thought you wanted it to be a robot.”
The above simple exchange is pretty good proof that the engineer is suffering from confirmation bias, IMO.
Related: The Eliza Effect.
* A pre-prompt warm up is a piece of text that "frames" the start of interaction with a large language model so that it can perform a specific task. At the start of any conversation, a paragraph of context is fed into the language model to set the statistics for a certain kind of document. Open AI's GPT-3 uses this a lot to support different tasks.
A simple pre-prompt might look like this:
The following is a conversation between a human and AI. The AI is knowledgable, friendly and helpful.
AI: Hi, I am AI! How can I help you today?
Human:
At that point control will pass to the human - outside of the AI - so they will add their first sentence that will become part of the growing prompt history. Note that the AI, if left to its own devices would begin to speak for the human too, naively continuing to predict the conversation unitil its end! In fact it does often do so, but the chatbot code is set to catch obvious end points such as newlines, or the AI actually outputting "Human: ", and will force the mode of conversation back to looking like a chat. It is really quite crude.
LaMDA's architecture may not do exactly the same as GPT-3, in fact I would suspect it is different in some key ways. However, it is very likely that there are still two "layers" in play - a very large sophisticated language model trained on gigabytes of human text, and a control layer which makes the language model conform to a chat scenario. Even if the control layer is more sophisticated than GPT-3's pre-prompting, it will have similar limitations.
