2
$\begingroup$

It has been mentioned in the research paper titled Generative Adversarial Nets that generator need to maximize the function $\log D(G(z))$ instead of minimizing $\log(1 −D(G(z)))$ since the former provides sufficient gradient than latter.

$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] + \mathbb{E}_{z ∼ p_z}[log (1 - D(z))]$$ In practice, the above equation may not provide sufficient gradient for $G$ to learn well. Early in learning, when $G$ is poor, $D$ can reject samples with high confidence because they are clearly different from the training data. In this case, $\log(1 −D(G(z)))$ saturates. Rather than training $G$ to minimize $\log(1 −D(G(z)))$ we can train G to maximize $\log D(G(z))$. This objective function results in the same fixed point of the dynamics of $G$ and $D$ but provides much stronger gradients early in learning.

A gradient is a vector containing the partial derivatives of outputs w.r.t inputs. At a particular point, the gradient is a vector of real numbers. These gradients are useful in the training phase by providing direction-related information and the magnitude of step in the opposite direction. This is my understanding regarding gradients.

What is meant by sufficient or strong gradient? Is it the norm of the gradient or some other measure on the gradient vector?

If possible, please show an example of strong and weak gradients with numbers so that I can quickly understand.

$\endgroup$
2
  • $\begingroup$ Technically, the term "sufficient graident" isn't quite well-defined. But usually what it means is the magnitude of the gradients are so small that it is unable to drive the optimization process. $\endgroup$
    – ayandas
    Aug 9, 2021 at 2:56
  • $\begingroup$ @AyanDas Does it mean they update weights very slowly or do not converge? $\endgroup$
    – hanugm
    Aug 9, 2021 at 2:58

1 Answer 1

1
$\begingroup$

The terms "insufficient gradient" or "not strong enough gradient" usually means that the magnitude of the gradient vector is too small or nearly zero that they can't drive the optimization properly.

Not having sufficient gradient is similar to having a very low learning rate - they are not only slow (in terms of convergence) but also drifts the optimization in a poor direction and get stuck in local minima.

$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .