I read Stephen Wolfram's piece explaining GPT which helped me a lot. I think what was most important was the idea that a Markov Model is fundamentally an unhelpful mental model for GPT type systems. While it is true that it works "like autocomplete", it is an autocomplete that is not based on simple probabilities between the sequences of words at all, but rather a system of hundreds of millions of programmatic neurons that organically adapted to find abstract patterns in text - not just patterns in word or letter frequencies, but patterns in which types of statement follow which other types, etc. In order to do this, multiple distinct processing steps seem to have developed organically within the hundreds of millions of neurons used. Wolfram explains that this system for example seems to have derived a theory of natural language syntax empirically from the input data. So what I was missing was a sense of the size of the model, and the idea that real semantic processing beyond mere word-probabilities was occurring, and how this type of processing could emerge from a system that was trained on mere word-by-word prediction.

I read Stephen Wolfram's piece explaining GPT which helped me a lot. I think what was most important was the idea that a Markov Model is fundamentally an unhelpful mental model for GPT type systems. While it is true that it works "like autocomplete", it is an autocomplete that is not based on simple probabilities between the sequences of words at all, but rather a system of hundreds of millions of programmatic neurons that organically adapted to find abstract patterns in text - not just patterns in word or letter frequencies, but patterns in which types of statement follow which other types, etc. 

In order to do this prediction, multiple distinct processing steps seem to have developed organically within the hundreds of millions of neurons used. Wolfram explains that this system for example seems to have derived a theory of natural language syntax empirically from the input data. At later stages, the system is probably doing analysis that we would consider to be "logical" or "conceptual" based on the fact data earlier language processing steps accomplished. 

So, what I was missing was a sense of the size of the model, and the idea that real semantic processing beyond mere word-probabilities was occurring, and how this type of processing could emerge from a system that was trained on mere word-by-word prediction.

I read Stephen Wolfram's piece explaining GPT which helped me a lot. I think what was most important was the idea that a Markov Model is fundamentally an unhelpful mental model for GPT type systems. While it is true that it works "like autocomplete", it is an autocomplete that is not based on simple probabilities between the sequences of words at all, but rather a system of hundreds of millions of programmatic neurons that organically adapted to find patterns in text - not just patterns in word or letter frequencies, but patterns in which types of statement follow which other types, etc. In order to do this, multiple distinct processing steps seem to have developed organically within the hundreds of millions of neurons used. Wolfram explains that this system for example seems to have derived a theory of natural language syntax empirically from the input data. So what I was missing was a sense of the size of the model, and the idea that real semantic processing beyond mere word-probabilities was occurring, and how this type of processing could emerge from a system that was trained on mere word-by-word prediction.

I read Stephen Wolfram's piece explaining GPT which helped me a lot. I think what was most important was the idea that a Markov Model is fundamentally an unhelpful mental model for GPT type systems. While it is true that it works "like autocomplete", it is an autocomplete that is not based on simple probabilities between the sequences of words at all, but rather a system of hundreds of millions of programmatic neurons that organically adapted to find abstract patterns in text - not just patterns in word or letter frequencies, but patterns in which types of statement follow which other types, etc. In order to do this, multiple distinct processing steps seem to have developed organically within the hundreds of millions of neurons used. Wolfram explains that this system for example seems to have derived a theory of natural language syntax empirically from the input data. So what I was missing was a sense of the size of the model, and the idea that real semantic processing beyond mere word-probabilities was occurring, and how this type of processing could emerge from a system that was trained on mere word-by-word prediction.