Suppose $G_{\phi}:\mathcal{Z}\rightarrow \mathcal{X}$ is a generator (neural network, non-invertible) that can sample from some distribution $\pi$ on $\mathcal{X}$. That is, $G_{\phi}(z)\sim \pi$ when $z\sim \mathcal{N}(0,I)$. Let $\phi+\delta_{\phi}$ represent a (small) perturbation of the parameters of $G_{\phi}$ and let $G_{\phi+\delta_{\phi}}(z)\sim \pi'$ when $z\sim \mathcal{N}(0,I)$.

Are there any results that quantify or bound $\mathcal{D}(\pi,\pi')$ in terms of $\delta_{\phi}$, where $\mathcal{D}$ is a distance measure for distributions (let's say KL-divergence, or the Wasserstein-1 distance)?

Basically, I want to know what kind of geometry is induced on the space of distributions by the Euclidian geometry on the parameter space of a generative adversarial network.

To explain further, let's consider a parametric family of distributions $p_{\phi}$, where $\phi\in\Phi$ (some parameter space). It is a fairly well-known result in statistics that $\text{KL}(p_{\phi}||p_{\phi+\delta_{\phi}})\approx \frac{1}{2}\delta_{\phi}^\top F_{\phi} \delta_{\phi}$, where $F_{\phi}$ is the Fisher information matrix. When the family $p_{\phi}$ is generated by a GAN with parameter $\phi$ (in which case we don't know $p_{\phi}$ in closed-form), can we have an analogous result?

  • $\begingroup$ I'm not really sure if I understood your question correctly, but here is some ideas. Let's say we have a 32x32 RGB image. It gives us $32^2\cdot 256^3$ dimensional space. But it turns out, that we can encode an image using latent features. When we talk about a distribution space in terms of GAN, we mean a lower-dimensional latent features manifold embedded in that higher-dimensional space. StyleGAN2 has done excellent research on this topic. $\endgroup$ Apr 2 '21 at 18:11
  • $\begingroup$ You can also take a look at this paper. The authors propose 50% dropdown layers that are used to generate images with some noise to make the model explore the underlaying manifold. $\endgroup$ Apr 2 '21 at 18:18
  • $\begingroup$ Please see the clarification I addded. Hope that clarifies my question. $\endgroup$
    – Subho
    Apr 2 '21 at 19:05
  • $\begingroup$ What do you mean by generating a probability distribution $p_{\phi}$ by a GAN? The GAN itself (namely the generator) is a probability distribution from which we can sample data (images). $\endgroup$ Apr 2 '21 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.