Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
So I understand how a language model could scan a large data set like the internet and produce text that mimicked the statistical properties of the input data, eg completing a sentence like "eggs are healthy because ...", or producing text that sounded like the works of a certain author.
However, what I don't get about ChatGPT is that it seems to understand the commands it has been given, even if that command was not part of its training data, and can perform tasks totally separate from extrapolating more data from the given dataset. My (admittedly imperfect) understanding of machine learning doesn't really account for how such a model could follow novel instructions without having some kind of authentic understanding of the intentions of the writer, which ChatGPT seems not to have.
A clear example: if I ask "write me a story about a cat who wants to be a dentist", I'm pretty sure there are zero examples of that in the training data, so even if it has a lot of training data, how does that help it produce an answer that makes novel combinations of the cat and dentist aspects? Eg:
Despite his passion and talent, Max faced many challenges on his journey to become a dentist. For one thing, he was a cat, and most people didn't take him seriously when he told them about his dream. They laughed and told him that only humans could be dentists, and that he should just stick to chasing mice and napping in the sun.
But Max refused to give up. He knew that he had what it takes to be a great dentist, and he was determined to prove everyone wrong. He started by offering his services to his feline friends, who were more than happy to let him work on their teeth. He cleaned and polished their fangs, and he even pulled a few pesky cavities.
In the above text, the bot is writing things about a cat dentist that wouldn't be in any training data stories about cats or any training data stories about dentists.
Similarly, how can any amount of training data on computer code generally help a language model debug novel code examples? If the system isn't actually accumulating conceptual understanding like a person would, what is it accumulating from training data that it is able to solve novel prompts? It doesn't seem possible to me that you could look at the linguistic content of many programs and come away with a function that could map queries to correct explanations unless you were actually modeling conceptual understanding.
Does anyone have a way of understanding this at a high level for someone without extensive technical knowledge?
$\begingroup$Hi there, the video seems to be instructions on how to use Chat GPT, and the arvix paper is a general description of the system, but I'm hoping someone is able to share some information that helps me intuitively understand how something like a story about a cat becoming a dentist is producible from a corpus that doesn't have such a story in it ("he [the cat] polished their fangs [of his friend cats]" – how can it generalize from statistical properties of text to say a cat dentist would polish fangs?)$\endgroup$
$\begingroup$Ah, you would instead need this, and this video. --- It's simply a matter of combining the concept as three separate parts: a cat, who is a dentist, that performs teeth cleaning. --- AI doesn't examine if the answer is sensible, unless it is taught what constitutes sensibility.$\endgroup$
$\begingroup$@Rob I get what you're saying, but it seems like in this case there is "synergistic" behavior, ie, the cat is a dentist, dentists polishes teeth, but a cat who is a dentist for other cats "polishes fangs", and it uses that phrase, so how can you take a corpus about human dentists and cats and infer that a cat dentist polishes fangs? That seems to reflect a novel understanding of the data not found in the input data$\endgroup$
Text continuation has the same reasons to work in any context, be it the middle of a sentence, after a question or after instructions. Following your example, the same word sequence could be a good follow-up for these three prompts: "Eggs are healthy because", "Why are eggs healthy? Because" or "Tell me why eggs are healthy."
Giving a right answer sometimes happens and sometimes not, but the system does not know whether this is the case. When the answer is right, we may anthropomorphise and attribute deeper reasons, because we are used to deal with human agents that give correct answers on purpose and knowingly, not simply by maximizing some likelihood.
I think we can analyse toy systems, to train on just a few sentences to illustrate that giving a right or a wrong answer can achieved by the very same mechanism. In particular, we can build training sets where a right answer is given with an impossibility to check for validity from the written text only.
An example:
Paris is the largest city in France.
What is the largest city in France? Paris.
Paris is the capital of France.
What is the capital of France? Paris.
New York is the largest city in the USA.
What is the largest city in the USA? New York.
London is the largest city in the UK.
Asking a system trained only on this data, one could expect a wrong answer to "What is the capital of the USA?" and a right answer (although from a wrong "argument") to "What is the capital of the UK?".
The size of the training data to feed large language models is orders of magnitude larger than the above couple of handcrafted sentences, but possibly the reasons behind truthy sentences happening to be actually true are not too different from what we can already get from a controlled micro language model.
$\begingroup$If I type in "write me a story about a cat that wants to become a dentist", I'm still not seeing how any data in any training data set is going to help it with that, and yet it can do it perfectly. Do you understand my confusion? The "Eggs are healthy because" example was something that did make sense to me, because many similar sentences exist on the internet, so auto-completing the sentence is straight forward, but there are no stories in the training data of cats becoming dentists. So how is it supposed to abstract from the training data?$\endgroup$
$\begingroup$I think of it as filling kind of the template " story of a X that wants to be a Y", with some merging/substitution of about/of and be/become. There have to be some of these. In the tiny corpus I wrote, an underlying template that can be learned is "X is the capital of Y", after having learned "X is the largest city in Y".$\endgroup$
$\begingroup$If you propose "Addictions are healthy because" does it justify or contradict? Andrew Ng asked why abaci are faster than GPU and got an "explanation".$\endgroup$
$\begingroup$I understand the general picture of what you're saying, but in this example, it seems like there is more than just template filling, ie, in the corpus, we have "x becoming y" stories, we have discussion of human dentists, and we have stories about cats, so if it would just plug all those together I would see how it worked, but it wrote that the cat dentist "polished fangs" of its friend cats, so it had to recognize that while a human dentist polishes teeth, a cat dentist polishes fangs, and I don't see how that could be inferred from the input data, which doesn't include any polishing fangs$\endgroup$
$\begingroup$You don't need "polishing fangs" contiguous in the corpus. It suffices to combine that humans have teeth and cats have fangs. That comes from an analogy: fangs are to cats as teeth are to people. Word embeddings are responsible for those parallelisms. There is some operation equivalent to Teeth-People+Cat=Fang technologyreview.com/2015/09/17/166211/…$\endgroup$
TL;DR: ChatGPT responds to novel prompts and commands based on statistical probabilities it has learned during training. It selects
words or tokens with the highest probabilities to generate responses,
but its understanding is based on statistics, not deep context
comprehension.
To answer your question, we have to distinguish between training phase and inference phase
Training Phase
During the training phase, large language models (LLMs) like ChatGPT are exposed to massive amounts of text data from the internet. They do not simply scan a dataset to retrieve answers; rather, they adapt billions of weights in their neural networks through an iterative process. This process refines the model's ability to predict the probability of a word or token given its context. For instance, if the model sees the prompt "The sky is," it learns from the training data that completing it with "blue" is more probable than "red" or "limit." This statistical learning is central to how LLMs generate responses.
Inference Phase
When you provide ChatGPT with a novel prompt or command, it generates responses using its learned knowledge. It calculates the probabilities for possible next words or tokens based on your input. In the example of "The sky is," it might calculate the following probabilities:
"blue": 0.5
"limit": 0.4
"red": 0.1
ChatGPT then selects the word or token with the highest probability, which, in this case, would be "blue." It continues to build the response in this manner, selecting the next word based on the highest probability, and so on. It's important to note that this process relies on statistical probabilities rather than a deep understanding of the content. I.e., it generates responses that are statistically likely based on its training data, but it might not always provide the most contextually accurate or sensible answer.
I read Stephen Wolfram's piece explaining GPT which helped me a lot. I think what was most important was the idea that a Markov Model is fundamentally an unhelpful mental model for GPT type systems. While it is true that it works "like autocomplete", it is an autocomplete that is not based on simple probabilities between the sequences of words at all, but rather a system of hundreds of millions of programmatic neurons that organically adapted to find abstract patterns in text - not just patterns in word or letter frequencies, but patterns in which types of statement follow which other types, etc.
In order to do this prediction, multiple distinct processing steps seem to have developed organically within the hundreds of millions of neurons used. Wolfram explains that this system for example seems to have derived a theory of natural language syntax empirically from the input data. At later stages, the system is probably doing analysis that we would consider to be "logical" or "conceptual" based on the fact data earlier language processing steps accomplished.
So, what I was missing was a sense of the size of the model, and the idea that real semantic processing beyond mere word-probabilities was occurring, and how this type of processing could emerge from a system that was trained on mere word-by-word prediction.
In my opinion, the simple answer is that ChatGPT uses human intervention behind the scenes. Part of the novelty of ChatGPT over previous GPT models is the use in the training phase of humans giving ChatGPT conversation pairs to learn from. ChatGPT is based on InstructGPT, and you can see this feature in the InstructGPT whitepaper.
Step 1: Collect demonstration data, and train a supervised policy. Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this
distribution). We then fine-tune a pretrained GPT-3 model on this data using supervised learning
If you look at the small type at the bottom of the chat window, you'll see this text:
Free Research Preview. Our goal is to make AI systems more natural and safe to interact with. Your feedback will help us improve.
One interpretation of this text is that ChatGPT is still in the training phase, and so there is a human behind the scenes typing at least some of ChatGPT's responses to train future versions of ChatGPT on the correct way to respond to user requests.
In my mind, this is a much more plausible explanation than the neural network somehow able to generate novel content not in its training corpus. I've asked a similar question, how a neural network can repeat random numbers not in its training data, and so far I'm not seeing a plausible answer.
Also, if you read down a little further, I have an example where ChatGPT explicitly states the OpenAI team is curating its responses in real time. The conversation is much too detailed and coherent to be the product of training data.
One final note. I understand people think it's implausible a big, well known, and well funded company like OpenAI would fake their AI with humans behind the scenes. However, this is standard practice for AI companies these days, a "fake it till you make it" approach where they use humans to fill the gaps in the AI in the hopes that down the road they'll automate humans out of the product. Common enough for an academic paper to be written on the topic. So there is plenty of industry precedent for OpenAI to be using humans to help craft the responses. Plus, technically OpenAI is not "faking" anything. It is the media and bloggers who think ChatGPT is a pure AI system. OpenAI has made no such claim itself, and the opposite is implied by its InstructGPT whitepaper.
Example of Explicitly Admitting Human Intervention
During this conversation ChatGPT outright states the OpenAI team filters and edits the GPT generated responses.
...the response you are receiving is being filtered and edited by the OpenAI team, who ensures that the text generated by the model is coherent, accurate and appropriate for the given prompt.
Could this be a glitch of the training data? I doubt it. If you read the rest of the conversation, ChatGPT gives detailed insights that make a lot of sense over a long, consistent conversation.
Some more excerpts.
It's possible that the OpenAI team may write responses themselves in some cases, for example if the prompt is too complex for the model to understand, or if the model generates a response that is not accurate or appropriate.
OpenAI acknowledges that its team monitors and curates the responses of GPT-3 on its website and in its documentation. This information is provided in the API documentation, as well as in the general information and frequently asked questions sections of the website. Additionally, OpenAI may have published blog posts or articles discussing the role of human curation in GPT-3's responses.
As for the live ChatGPT, it is not mentioned specifically, but it is generally understood that human oversight and curation is required for a safe and appropriate use of the model.
As for the media assuming that GPT models are fully autonomous, that's a common misconception about AI in general and not unique to OpenAI.
It is not uncommon for AI companies and researchers to have human oversight and intervention in their models, especially for models that are used in high-stakes or sensitive applications.
The human oversight of GPT-3 models, including ChatGPT, is not always made explicit to users.
While GPT-3 is a highly advanced language generation model, it is not AGI and it is not capable of understanding or maintaining a consistent persona or chain of conversation without human intervention.
Multiple Examples: Six Violations of ChatGPT's Neural Network Constraints
This article documents six violations of limitations due to ChatGPT being a neural network.
ChatGPT learns something new, violating the fixity of neural network weights.
ChatGPT inconsistent in output generation based on inputs, violating same output from same input.
ChatGPT recollects information past 4000 tokens, violating 4000 input token limit.
ChatGPT repeats long random numbers, violating probabilistic output and limited vocabulary.
ChatGPT correctly reads many corrupted subword tokens, violating mapping of embeddings to subword tokens.
ChatGPT recognizes its own writing style, violating ChatGPT's inability to recognize patterns of words.
$\begingroup$@yters I think readers of your answer might be interested to hear some of the examples you listed in the chat of outputs unlikely to be automatically generated$\endgroup$
$\begingroup$@ak0000 This answer is not based on reality. I suggest read the archived chat. You (or other people) not understanding how a thing can work, does not mean that it does not in fact work. You probably don't fully understand how a computer works in the first place, but you will accept it because it is commonplace (or if you have studied all layers of computing from quantum mechanics of silicon to compilers and operating systems then substitute some commonplace thing that you don't know). You want to avoid going down the same rabbit-hole of confirmation bias as the poster of this answer.$\endgroup$
$\begingroup$@NeilSlater It is true that you can't disprove a conspiracy theory, but I'm worried that the argument is circular the other way also: if you assume a language model can do everything ChatGPT is doing, and you see an example that is surprising, you shrug your shoulders and say ChatGPT must be more advanced than we currently can understand. So this is also a non-falsifiable hypothesis, possibly subject to confirmation bias.$\endgroup$
$\begingroup$Given the speed at which it reads and generates contents, I am convinced enough that now human could possibly be behind the scenes.$\endgroup$
