How does:
$$\text{Var}(y) \approx \sigma^2 + \frac{1}{T}\sum_{t=1}^Tf^{\hat{W_t}}(x)^Tf^{\hat{W_t}}(x_t)-E(y)^TE(y)$$ approximate variance?

I'm currently reading What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision, and the authors wrote the above formula for the approximate estimation for the variance. I'm confused how the above is an approximation for $\frac{\sum(y-\bar{y})^2}{N-1}$. So, in the above equation, they're using a Bayesian Neural Network to quantify uncertainty. $\sigma$ is the predictive variance (kind of confused how they get this). $x$ is the input and $y$ is the label for the classification. $f^{\hat{W_t}}(\cdot)$ output a mean to a Gaussian distribution, with $\sigma$ being the SD for that distribution and $T$ is a predefined number of samples because the gradient is evaluated using Monte Carlo sampling.

  • $\begingroup$ Can you add the dimensions of all the matrices here? It kind of seems a matrix+real number addition. $\endgroup$ – DuttaA Feb 20 at 5:14
  • $\begingroup$ @DuttaA sadly they don't give dimensions. I think the reason being you can adjust the model size to fit whatever you need, so there's no strict requirement in terms of dimensionality. If you like you can just think of them as vectors I think that's reasonable. $\endgroup$ – user8714896 Feb 20 at 5:15
  • $\begingroup$ Hmm,,,the notations then kind of don't make sense...the 1st term is a real..second term is a matrix and the 3rd term is real again. $\endgroup$ – DuttaA Feb 20 at 5:16
  • $\begingroup$ dot product maybe? $\endgroup$ – user8714896 Feb 20 at 5:19
  • 1
    $\begingroup$ @nbro go ahead and put an answer. I posted on stats stack exchange to see if stats people had any intuition about this, that's why I deleted it, but I'm eagerly waiting to see what you write up. $\endgroup$ – user8714896 Feb 21 at 0:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.