# Comparison of the two alternative forms for the KL divergence [closed]

On page 468 of 'Pattern Recognition and Machine Learning', what does 'the same variables given by the product of two independent univariate Gaussian distributions' mean?

The PDF says,

The green contours corresponding to 1, 2, and 3 standard deviations for a correlated Gaussian distribution p(z) over two variables z1 and z2, and the red contours represent the corresponding levels for an approximating distribution q(z) over the same variables given by the product of two independent univariate Gaussian distributions whose parameters are obtained by minimization of (a) the KullbackLeibler divergence KL(q||p), and (b) the reverse Kullback-Leibler divergence KL(p||q).

but I don't understand the explanation clearly.

Which is the same variables?
Which is the two distributions?
What is the product? Does it mean the probability density function?
Thanks.

• Can you please ask a specific question? "I don't understand the explanation clearly." is not a question and does make clear what you don't understand. Please, once you've decide what your specific question is, put it also in the title. Thanks.
– nbro
Commented Jul 20, 2023 at 13:04
• I added specific questions such as 'same variables' and 'two distributions'. Thanks. Commented Jul 20, 2023 at 14:28

The KL divergence is just not symmetric, and so changing $$q$$ for $$p$$, and vice-versa, gives you a different behavior because the expectation is computed on a different distribution.

• In the first plot, you see the forward KL, $$KL(q\ \|\ p)$$, that encourages the approximating distribution $$q(z)$$ (the one you topically learn) to cover the mode of $$p(z)$$: this is called mode-seeking behavior, which tries to match the most-likely point of the distribution $$p$$.
• The other plot shows, instead, the reverse KL, $$KL(p\ \|\ q)$$, that tries to match the support of $$p(z)$$, however loosing the mode in the approximating distribution.

To recap, the forward KL tries to put probability mass in the area of the distribution where the probability of samples is higher (in this example, the area where the std is 1), while the reverse KL tries just to cover all the distribution's support placing some non-zero probability everywhere: that's why the second, bigger, red circle covers the most of the green contours.

Update:

Which is the same variables?

The (random) variables $$z_1$$ and $$z_2$$ define the "space" of values that you can observe, e.g. you're given a dataset of samples taken from $$p(z)$$, i.e. the green distribution, where $$z \in \mathbb [0,1]^2$$. Basically, the approximating distribution, $$q(z)$$, is defined on the same space where the variables $$z_1$$ and $$z_2$$ lie.

Which is the two distributions?

I think Bishop assumes $$q(z)$$ to be defined as a mean field approximation, which should correspond to $$q(z) = \prod_{i\in\{1,2\}} q(z_i; \mu_i, \sigma)$$ where $$\mu_i$$ and $$\sigma$$ are the parameters you want to learn to approximate $$p(z)$$. Basically, you consider two tractable distributions, e.g. a Gaussian, and learn the parameters of each "simple" distribution. Then, to get the final, more complex, distribution you perform the mean field assumption which is about doing a product among each $$q_i(z_i)$$. Also I put $$\sigma$$ and not $$\sigma_i$$ since from the figures the variance seems to be independent from $$z_i$$. For example, the mean field assumption does not enable you to learn the correlation between each $$z_i$$, so this may explain the same variance across dimensions.

What is the product? Does it mean the probability density function?

I think a partial answer is above. Anyway, is a product over the pdf of each $$p_i$$ and $$q_i$$ to yield, respectively, $$p(z)$$ and $$q(z)$$. But in the case of $$p(z)$$ you also account for the correlations when doing the product, instead in $$q(z)$$ independence is assumed.

In the end, you can define $$p(z) = \mathcal N(\mu,\Sigma)$$ where $$\mu=[0.5,0.5]$$ and $$\Sigma=\begin{bmatrix} \sigma_{11} & \sigma_{12} \\ \sigma_{21} & \sigma_{22} \end{bmatrix}$$, where each $$\sigma_{ij}$$ (for $$i=1,2$$ and $$j=1,2$$) tells you the correlation between variable $$z_i$$ and $$z_j$$ that has the effect of shrinking or expanding and rotating (more importantly) the green Gaussian. Instead, $$q(z)=\prod_i \mathcal N(\hat\mu_i,\hat\sigma_i)$$, since is an approximation, has no such capabilities (e.g., rotation) and so you get a perfect circle, smaller or larger according to the KL you minimize to learn the parameters: $$\hat\mu_1 = \hat\mu_2 = 0.5$$ and $$\hat\sigma_1 = \hat\sigma_2 = \hat \sigma$$ or $$\hat\Sigma=\begin{bmatrix} \hat\sigma & 0 \\ 0 & \hat\sigma \end{bmatrix}$$.

• @diffusionstable I'm glad the answer helped you. I've edited it trying to answer the other questions too. Have a look and let me know! Commented Jul 21, 2023 at 19:22
• Thank you. It's a bit complicated, so I'll try to read it slowly! Commented Jul 22, 2023 at 14:30