0
$\begingroup$

How to expand reconstruction error to mean squared error when it is $\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$?

[reconstruction error]
$\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$

[mean square error]
$\mathbb{E} \Big [\big(x - p_\theta(q_\phi(x))\big)^2 \Big]$

[MSE pseudo code]

def reconstruction_loss(y, t):
    # MSE y:predicted value
    #     t:true value
    return square(y - t).mean()

Is the reconstruction error just an idea and is actual formula the MSE? Is that all?

I mean, why does MSE (or BCE) comes from reconstruction error?

$\endgroup$
6
  • $\begingroup$ I see 2 problems/questions here (reconstruction to MSE and reconstruction to BCE). Could you please focus on 1 problem at a time? You should create a separate post for the 2nd question. Thanks. You should also provide more details about why you want to do this and why you're stuck at doing it or what your ideas are. $\endgroup$
    – nbro
    Aug 3, 2023 at 13:20
  • $\begingroup$ I edited the question clearly. Thank you. $\endgroup$ Aug 3, 2023 at 17:12
  • $\begingroup$ Thanks. Can you please now provide the source of the MSE you're showing us? This isn't from the original paper. Something doesn't seem to be right in that formula. $\endgroup$
    – nbro
    Aug 4, 2023 at 0:42
  • $\begingroup$ Yes. I Added pseudo code of MSE. $\endgroup$ Aug 4, 2023 at 7:43
  • $\begingroup$ Yes, the formula is not consistent with the pseudocode. The parentheses are not correctly placed. You need to square the subtraction according to the pseudocode $\endgroup$
    – nbro
    Aug 4, 2023 at 10:29

2 Answers 2

3
$\begingroup$

In a way, you're right. The reconstruction loss is just an idea because you have not yet defined the distribution $p_\theta$. If you assume that this distribution is e.g. a Gaussian, then you should be able to derive something interesting. See equation 12 from the VAE paper. You can check out the definition of the Gaussian pdf on Wikipedia. If you're stuck, you could start with the simple regression problem and, if necessary, watch e.g. this Hinton's video.

$\endgroup$
2
$\begingroup$

If you model the distribution $p_\theta$ as Gaussian then:

$\displaystyle p_\theta(x|z) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\frac{x-\mu}{\sigma}^2} $

$\displaystyle \log p_\theta(x|z) = -log \sigma - \frac{1}{2}\log 2\pi -\frac{1}{2} \frac{x-\mu}{\sigma}^2$

If you decide that your neural net will only produce the mean $\mu$ and you will use a constant $\sigma$, then the derivative of $\log p_\theta$ is the same as the derivative of $MSE$, but scaled with $\sigma$.

Now, given that this is the most common choice of model (Gaussian with constant $\sigma$), people just optimize the $MSE$ and adjust their learning rate accordingly.

$\endgroup$
1
  • 1
    $\begingroup$ Is that $ - \log\sigma - \frac{1}{2} \log 2\pi - \frac{1}{2} \Big( \frac{x-\mu}{\sigma} \Big)^2 $ ? $\endgroup$ Aug 4, 2023 at 14:22

Not the answer you're looking for? Browse other questions tagged .