Parametric vs Non-parametric generative models

Question

I have a little perplexity trying to distinguish parametric vs non-parametric generative model.

In my understanding, a parametric generative model would try to learn the probability density function by estimating the parameters of an underlying distribution we are assuming. So just doing for example,

$$\theta^* = arg\max_\theta \,\prod_{i=1}^N p_\theta(\textbf{x}_i)$$

I realize that in practice, we need to figure out what is the basic distribution that we are going to modify by adjusting the parameters $\theta$. So in the case of VAEs we use latent variables assumption to make training feasible, we jointly train $q_\phi(\textbf{z}|\textbf{x})$ and $p_\theta(\textbf{x}|\textbf{z})$ prametrizing both distributions with neural networks (i.e. encoder & decoder). In such case, we end up with the situation that all our distributions are Gaussians (assuming that prior and conditional are gaussians). So, having said that, can we conclude that VAEs are parametric? Also, what could be an example of non-parametric generative model?

I would say that, for example, GANs maybe an example of non-parametric model, as we start with a latent normal distribution but then applying a stack of non-linear transformations ending up with something potentially very complicated.

Answer 1

I call this difference prescribed vs implicit models.

To my knowledge, a parametric model has a fixed,finite number of parameters with respect to the sample size (e.g a linear model), while nonparametric model have an increasing number of parameters with respect to data. See this blog post and this answer for more precise info. Since most models use a Neural Network to generate an output, I prefer the terms prescribed vs implicit.

Viewing this way,

• Implicit models directly generate the output by passing a noise vector through a deterministic function (Generally a NN). They do not have a global likelihood, and are less limited in the architecture. A great example of an implicit model, as you stated, are GANs.

• Prescribed models provides an explicit parametric specification of the output distribution $p(x)$. They can estimate the likelihood of data, but are more constrained in the architecture.

  • An Autoregressive model fits a Bernoulli distribution , learning a neural network to model $p(x_i|x_{<i})$
  • Normalizing flows learn a change of variable to reshape a gaussian source noise. They must learn an invertible function, so the architecture is heavily constrained.
  • VAEs assume a Gaussian prior $p(z) \sim \mathcal{N}(0,\mathbb{I})$ that is decoded by a NN which models $p(x|z)$. They have less limits in the architecture, but they lose the closed form likelihood, and they are trained by maximizing an ELBO. Since they still have (even if approximated) likelihood, they are prescribed models.