Is there any difference between the model distribution and data distribution, or are they the same?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Yes. In Machine Learning we consider that the samples in your training set are sampled from an underlying distribution called the data generating distribution.
Generative models classify the samples by trying to learn the distribution of the data. In most cases, either the model is incapable of doing so, or the training samples aren't enough to properly describe the data-generating distribution, so the model learns an approximation of this. This is what you call the model's distribution.
You can find more info about these concepts in a more detailed answer I wrote. If you're familiar with GANs, you can also read this post, to see where these two concepts come into play when training the two networks.
-
$\begingroup$ This answer seems correct, but why are you talking about generative models? Are model distributions only associated with generative models? Is the terminology "data distribution" and "model distributions" only useful in the case of GANs or generative models? Is this terminology widely known in machine learning, or only Goodfellow's book uses it? $\endgroup$– nbroCommented Apr 5, 2020 at 14:09
-
1$\begingroup$ Generative models is a broad category of ML models including the generator from GANs, naive bayes, etc. The other category are discriminative models, which don't try to explicitly model the data's distribution, i.e. there's no notion of a model's distribution. The terms "generative" and "discriminative" models are widely used, as is the notion of the data-generating distribution. I'm unaware, though, if the latter was coined by Goodfellow or not (it might have also different names). Nowadays, though, it is the only widely-used terminology I'm aware of. $\endgroup$– Djib2011Commented Apr 5, 2020 at 15:27