1
$\begingroup$

I was going through the paper on GAN by Ian Goodfellow. Under the related work section, there is an equation. I cannot decipher the equation. Can anyone help me understand the meaning of the equation?

$$\lim_{\sigma \to 0} \nabla_{\mathbf x} \mathbb E_{\epsilon \sim \mathcal N(0, \sigma^2 \mathbf I)} f(\mathbf x+\epsilon) = \nabla_x f(\mathbf x)$$

Also, any guide to understanding mathematical notation for reading research paper is highly appreciated.

$\endgroup$
2
$\begingroup$

In full:

The limit, as standard deviation $\sigma$ tends towards zero, of the gradient with respect to vector $\mathbf{x}$, of the expectation - where perturbation $\epsilon$ follows the normal distribution with mean 0 and variance $\sigma^2$ times identity vector $[1,1,1,1...]$ * - of any function $f$ of $\mathbf{x}$ plus $\epsilon$ is equal to the gradient with respect to $x$ of the same function of $\mathbf{x}$.

If we break that down:

$$\lim\limits_{\sigma \rightarrow 0}$$

The limit, as standard deviation $\sigma$ tends towards zero of

$$\nabla_\mathbf{x}$$

the gradient with respect to vector $\mathbf{x}$ of

$$\mathbb{E}$$

the expectation ...

$$\mathbb{E}_{\epsilon \sim \mathcal{N}(0, \sigma^2\mathbf{I})}$$

[the expectation] - where perturbation $\epsilon$ follows the normal distribution with mean 0 and variance $\sigma^2$ times identity vector $[1,1,1,1...]$ * - of

$$f(\mathbf{x} +\epsilon)$$

any function $f$ of $x$ plus $\epsilon$

$$ = \nabla_x f(\mathbf{x})$$

is equal to the gradient with respect to $\mathbf{x}$ of the same function $f(\mathbf{x})$.

Basically it says that taking small perturbations of a vector input to a function and measuring the gradient at those varied points can be used to give you a valid estimate of the true gradient at the point you are making variations of.

In terms of understanding the equations, read introductory texts to the research area, and if like me your maths has been unused for many years before attempting this, expect to spend time and effort. Re-read the equations, memorise and write out the basic ones from the field, apply them to simple problems that might be presented in text books. Reading maths equations is not much different to reading music or reading another language - it takes concentration, practice, time and effort to become fluent enough to read an equation and comprehend it. The different research fields can be quite different too, some might be as similar enough to get by with what you know already, others may require learning all over again.


* I am not 100% certain of the interpretation of $\mathbf{I}$ as an identity vector - a matrix may be more appropriate, which depends on the form of $\mathcal{N}(\mu, \sigma^2)$ when handling vector distributions. A matrix form for the second argument would be more general and allow for covariance, although the use of $\mathbf{I}$ would then explicitly remove covariance and make each component of $\epsilon$ independent, which is required for this result.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Are you sure about $I$ being $[1,1,...]$ though? In general we know it stands for identity vector and I think this notation is especially important to show that the variables are independent which results in covariance being 0 $\endgroup$ – DuttaA Apr 4 at 12:51
  • $\begingroup$ @DuttaA: Actually I am not sure, it depends on the form of second argument $\mathcal{N}$, which I am assuming will just take a vector of variances for independent vector components here, but you are right it could also support some forms of covariance in general, and we'd want an identity matrix instead to make the variables independent. Do you have a reference to the common assumptions of that function? $\endgroup$ – Neil Slater Apr 4 at 13:13
  • $\begingroup$ In general GANs probably come under somewhat probabilistic graphical models so I thought it might be the case. But I'll check and let you know if I come across something concrete. $\endgroup$ – DuttaA Apr 4 at 13:17
  • $\begingroup$ arxiv.org/pdf/1312.6114.pdf check this paper section 3, it is similar in purpose according to the author for the aforementioned equation. Especially the text above eqn 9. $\endgroup$ – DuttaA Apr 4 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.