Why are logarithms used in GANs minimax equation?

Question

The minimax equation for generative adversarial networks

$$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[\log D(\boldsymbol{x})] + \mathbb{E}_{\boldsymbol{z}\sim p_{\boldsymbol{z}}(\boldsymbol{z})}[\log(1 - D(G(\boldsymbol{z}))] $$

Why do we use logarithms instead of just

$$\min_G \max_D V(D,G) = \mathbb{E}_{\boldsymbol{x}\sim p_{data}(\boldsymbol{x})}[ D(\boldsymbol{x})] + \mathbb{E}_{\boldsymbol{z}\sim p_{\boldsymbol{z}}(\boldsymbol{z})}[(1 - D(G(\boldsymbol{z}))] $$

Answer 1

It's common in machine learning to do this log-trick, i.e. rather than optimizing $f(\mathbf{x})$, you optimize $\log f(\mathbf{x})$.

There are 3 main reasons why we (can) do this.

1. When your objective function is the product of multiple probabilities (or, more generally, small numbers), i.e. $f(\mathbf{x}) = \prod_{i=1}^N p(x_i)$, then $\log f(\mathbf{x}) = \log \left( \prod_{i=1}^N p(x_i) \right) = \sum_{i=1}^N \log p(x_i)$ (see this), which is more numerical stable because we got rid of multiplication of possibly very small numbers, which can lead to underflow.

2. For the same reason, we can also compute derivatives more easily, as we can simply compute the derivatives of each component (because derivatives are linear)

3. The logarithm is monotonically increasing, so $\log f(\mathbf{x})$ has the same optima as $f(\mathbf{x})$ (simple proof here).

Answer 2

Because Binary Cross-Entropy Loss is a commonly used loss function for binary classification problems in machine learning:

$$ L = - \frac{1}{N} \sum_{i=1}^{N} \left[ y_i \cdot \log(\hat{y_i}) + (1 - y_i) \cdot \log(1 - \hat{y_i}) \right] $$

• $N$ is the number of samples.
• $y_i$ is the true label of the $i$-th sample (0 or 1).
• $\hat{y_i}$ is the predicted probability for the $i$-th sample (i.e., the probability of predicting 1).

If the true label $y_i=0$, the loss is $\log(1 - \hat{y}_i)$, and similarly, if $y_i=1$, the loss is $\log(\hat{y}_i)$.

By dividing the samples based on their true labels, we arrive at the form used in GANs:

$$ \mathbb{E}_{\boldsymbol{x} \sim p_{\text {data }}(\boldsymbol{x})}[\log D(\boldsymbol{x})]+\mathbb{E}_{\boldsymbol{z} \sim p_{\boldsymbol{z}}(\boldsymbol{z})}[\log (1-D(G(\boldsymbol{z})))] $$

As for why Binary Cross-Entropy Loss is chosen for binary classification problems, that is a broader question.

Additionally, it's important to note that GANs do not only train the discriminator but also the generator. The choice of such a joint optimization objective can be explained by the fact that the generator's optimization goal is to minimize the Jensen-Shannon (JS) divergence between the real data distribution $p_{\text{data}}$ and the generated data distribution $p_g$.

The original paper proves that when optimizing the discriminator $D$, fixing $G$, it becomes a maximization problem. The optimal solution for the discriminator is:

$$ D^*_G(\boldsymbol{x}) = \frac{p_{\text{data}}(\boldsymbol{x})}{p_{\text{data}}(\boldsymbol{x}) + p_g(\boldsymbol{x})} $$

Substituting $D^*_G$ into $V(D, G)$ gives:

$$ V(D^*_G, G) = \mathbb{E}_{\boldsymbol{x} \sim p_{\text{data}}} \left[ \log \frac{p_{\text{data}}(\boldsymbol{x})}{p_{\text{data}}(\boldsymbol{x}) + p_g(\boldsymbol{x})} \right] + \mathbb{E}_{\boldsymbol{x} \sim p_g} \left[ \log \frac{p_g(\boldsymbol{x})}{p_{\text{data}}(\boldsymbol{x}) + p_g(\boldsymbol{x})} \right] $$

This simplifies to:

$$ V(D^*_G, G) = -\log 4 + 2 \cdot JS(p_{\text{data}} \| p_g) $$

Therefore, the optimization objective $\min_G V(D^*_G, G)$ is equivalent to minimizing the JS divergence:

$$ \min_G JS(p_{\text{data}} \| p_g) $$

However, with the alternative optimization objective you mentioned, after a similar derivation, we find:

$$ V'(D^*, G) = \mathbb{E}_{\boldsymbol{x} \sim p_{\text{data}}} \left[ \frac{p_{\text{data}}}{p_{\text{data}} + p_g} \right] + \mathbb{E}_{\boldsymbol{x} \sim p_g} \left[ \frac{p_g}{p_{\text{data}} + p_g} \right] = 1 $$

Unlike the original GAN, the alternative loss function does not directly express a divergence measure between $p_{\text{data}}$ and $p_g$. The original GAN provides a symmetric and meaningful distribution alignment measure through JS divergence, while the alternative loss function lacks this theoretical foundation and cannot guarantee that $p_{\text{data}} = p_g$ at the optimum.