How does:
$$\text{Var}(y) \approx \sigma^2 + \frac{1}{T}\sum_{t=1}^Tf^{\hat{W_t}}(x)^Tf^{\hat{W_t}}(x_t)-E(y)^TE(y)$$ approximate variance?

I'm currently reading What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision, and the authors wrote the above formula for the approximate estimation for the variance. I'm confused how the above is an approximation for $\frac{\sum(y-\bar{y})^2}{N-1}$. So, in the above equation, they're using a Bayesian Neural Network to quantify uncertainty. $\sigma$ is the predictive variance (kind of confused how they get this). $x$ is the input and $y$ is the label for the classification. $f^{\hat{W_t}}(\cdot)$ output a mean to a Gaussian distribution, with $\sigma$ being the SD for that distribution and $T$ is a predefined number of samples because the gradient is evaluated using Monte Carlo sampling.

  • $\begingroup$ Can you add the dimensions of all the matrices here? It kind of seems a matrix+real number addition. $\endgroup$
    – user9947
    Feb 20, 2021 at 5:14
  • $\begingroup$ @DuttaA sadly they don't give dimensions. I think the reason being you can adjust the model size to fit whatever you need, so there's no strict requirement in terms of dimensionality. If you like you can just think of them as vectors I think that's reasonable. $\endgroup$ Feb 20, 2021 at 5:15
  • $\begingroup$ Hmm,,,the notations then kind of don't make sense...the 1st term is a real..second term is a matrix and the 3rd term is real again. $\endgroup$
    – user9947
    Feb 20, 2021 at 5:16
  • $\begingroup$ dot product maybe? $\endgroup$ Feb 20, 2021 at 5:19
  • 1
    $\begingroup$ @nbro go ahead and put an answer. I posted on stats stack exchange to see if stats people had any intuition about this, that's why I deleted it, but I'm eagerly waiting to see what you write up. $\endgroup$ Feb 21, 2021 at 0:03


You must log in to answer this question.