1
$\begingroup$

I'm working my way through how ChatGPT works. So I read that ChatGPT is a generative model. When searching for generative models, I found two defintions:

  • A generative model includes the distribution of the data itself, and tells you how likely a given example is
  • Generative artificial intelligence (AI) describes algorithms (such as ChatGPT) that can be used to create new content

Do they both mean the same? That is, for generating new content, a model must learn the distribution of data itself? Or do we call chatGPT generative because it just generates new text? I see that ChatGPT is something other than a discriminative model that learns a boundary to split data, however, I can not bring ChatGPT in line with a more traditional generative model like naive bayes, where class distributions are inferred.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

What are generative (and discriminative) models?

If the model learns a distribution of the form $p(x)$ or $p(x, y)$, where $x$ are the inputs and $y$ the outputs/labels, from which you can sample data, then it's a generative model. An example of a generative model: variational autoencoder (VAE).

Bishop also defines generative models in this way (p. 43)

Approaches that explicitly or implicitly model the distribution of inputs as well as outputs are known as generative models, because by sampling from them it is possible to generate synthetic data points in the input space

If it learns a distribution of the form $p(y \mid x)$, then it's a discriminative model - many/most classifiers learn this distribution, but you can also derive the conditional given the the joint and prior (that's why above Bishop uses implicitly or explicitly).

Bishop also defines discriminative models in this way (p. 43)

Approaches that model the posterior probabilities directly are called discriminative models

The related Wikipedia article claims that people have not always been using these terms consistently (which is common in machine learning), so one should always keep that in mind.

GPTs are autoregressive

As far as I know, GPTs are autoregressive models. Here is another potentially useful post that explains what autoregressive models are.

My understanding of autoregressive models, at least based on neural networks, is that they are also generative models - the linked articles and even the GPT-2 paper seem to start the descriptions from the assumption that you can factorize some joint distribution like $p(x)$ into conditional distributions.

ChatGPT is based on a GPT model, so it's probably considered a generative model too, but there are several steps involved to create this model, so it may not be super clear how to categorise this model.

Moreover, the authors of the transformer, which GPT models are based on, claim that the transformer is an autoregressive model.

Conclusion

It seems to me that many people in ML refer to any model that generates data as a generative model, even if there's no written theoretical formulation of it as a generative model, which doesn't mean that you cannot formulate these models as generative models, i.e. a model that learns some distribution that you can use to sample data from data distribution.

I am currently not familiar enough with the details of the GPT models to say if they have been mathematically formulated as generative models of the form $p(x, y)$, but they model some distribution of the form $p(x)$, from which you can sample, otherwise, how could you even sample data (words)?

Improve this answer
$\endgroup$
6
  • $\begingroup$ from the wikipedia article you linked to, a generative model must model $p(x,y)$ so why a model learns only $p(x)$ is also generative? $\endgroup$
    – Sam
    Sep 1 at 21:50
  • $\begingroup$ You can define $x = z, y$, for example. Or $x = (x_1, x_2, x_3, ..., x_n)$, or $x = x_1$. In other words, you can have a distribution over many variables. In my answer, I actually say $p(x)$ or $p(x, y)$ in the first sentence $\endgroup$
    – nbro
    Sep 1 at 21:53
  • $\begingroup$ I feel that is still a bit rhetorical... the way I understand it is, a generative model must be able to model $p(x,y) = p(y|x)p(x)$. So here $p(x)$ is not a joint but indeed a prior over $x$. In the case of VAE, it is clear to me how it models it (like you said in this case $x$ is $z$), but it is not clear to me how do GPT-series models model this distribution. $\endgroup$
    – Sam
    Sep 2 at 22:14
  • $\begingroup$ @Sam Let's say you have $p(x, y) = p(y \mid x) p(x)$. Now, let $x = z, w$, so $p(x, y) = p(y \mid x) p(x) = p(y \mid z, w) p(z, w)$. Now, $p(z, w)$ is the prior, but it's a joint? Yes. You can do a similar thing for the evidence (denominator). Now, let $x = x_1$ and $y = x_2$, then $p(x, y) = p(x_1, x_2) = p(x_1 \mid x_2) p(x_2)$. Now, in the case of models like GPTs, you probably can model them as models that produce $x_{t+1}$ given previous tokens. So, in a way, they might model kind of a joint distribution over the current token and the previous ones. $\endgroup$
    – nbro
    Sep 4 at 11:37
  • $\begingroup$ To be honest, I don't know how they were originally formulated (I didn't have the chance to read the papers yet), but what I am trying to say is that $p(x)$ is not necessarily a distribution over just one variable. You can do some kind of algebra with these probabilities and changes of variables, etc. $\endgroup$
    – nbro
    Sep 4 at 11:37
2
$\begingroup$

They both refer to the same type of models. However, the second definition is a more 'intuitive' explanation of what generative AI does, while the first is a definition that refers more to what a generative model is.

To generate new data similar to some training data (definition 2), a model needs to learn the training data distributions (definition 1). Only if the model has learned that distribution it can use that distribution to sample (generate) new data from that distribution.

During training, ChatGPT also learned the distribution of the training data that OpenAI provided the model with. After training, the model simply takes in the input and uses the input to sample from the learned distribution to generate an output. So ChatGPT also follows both of your definitions.

Improve this answer
$\endgroup$
5
  • $\begingroup$ is it the distribution over, "what word is most probable when seeing this part of the sentence"? Or how would you discribe over what the distribution is learned? $\endgroup$
    – Ai4l2s
    Feb 2 at 9:16
  • 1
    $\begingroup$ Its exactly that for large language models (LLMs) $\endgroup$
    – Robin van Hoorn
    Feb 2 at 10:00
  • $\begingroup$ is a Vanilla RNN that predicts time series data then also considered as a generative model? $\endgroup$
    – Ai4l2s
    Feb 2 at 11:26
  • $\begingroup$ Prediction models can be seen as a broader category of models of which generative models could be a 'subgroup'. Prediction models can also 'predict the class' of some data instance. I guess ChatGPT can be seen as a prediction model as well to some extend. $\endgroup$
    – Robin van Hoorn
    Feb 2 at 12:24
  • $\begingroup$ An RNN that predicts a time series can be seen as 'generating the future of the time-series'. Definitions of model-groups are never clear cut. $\endgroup$
    – Robin van Hoorn
    Feb 2 at 12:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .