How to expand reconstruction error to mean squared error in Variational AutoEncoder? [closed]

Question

How to expand reconstruction error to mean squared error when it is $\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$?

[reconstruction error]
$\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$

[mean square error]
$\mathbb{E} \Big [\big(x - p_\theta(q_\phi(x))\big)^2 \Big]$

[MSE pseudo code]

def reconstruction_loss(y, t):
    # MSE y:predicted value
    #     t:true value
    return square(y - t).mean()

Is the reconstruction error just an idea and is actual formula the MSE? Is that all?

I mean, why does MSE (or BCE) comes from reconstruction error?

Answer 1

In a way, you're right. The reconstruction loss is just an idea because you have not yet defined the distribution $p_\theta$. If you assume that this distribution is e.g. a Gaussian, then you should be able to derive something interesting. See equation 12 from the VAE paper. You can check out the definition of the Gaussian pdf on Wikipedia. If you're stuck, you could start with the simple regression problem and, if necessary, watch e.g. this Hinton's video.

Answer 2

If you model the distribution $p_\theta$ as Gaussian then:

$\displaystyle p_\theta(x|z) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\frac{x-\mu}{\sigma}^2} $

$\displaystyle \log p_\theta(x|z) = -log \sigma - \frac{1}{2}\log 2\pi -\frac{1}{2} \frac{x-\mu}{\sigma}^2$

If you decide that your neural net will only produce the mean $\mu$ and you will use a constant $\sigma$, then the derivative of $\log p_\theta$ is the same as the derivative of $MSE$, but scaled with $\sigma$.

Now, given that this is the most common choice of model (Gaussian with constant $\sigma$), people just optimize the $MSE$ and adjust their learning rate accordingly.