What is uncentered variance and how it becomes equal to mean square in Adam?

Question

I have been reading about Adam and AdamW (Here). The author mentioned that in "uncentered variance" we don't consider subtracting mean

In this statement, the author is talking about uncentered variance and how it becomes equal to the square of the mean.

I want to understand what exactly is uncentered variance? (because if I consider the general equation of variance $\dfrac{(\text{obs-mean})^2}{(n-1)}$ then it is not making sense to me the removing mean will lead to the definition of uncentered variance as we need a point around which we are calculating variance and here mean is that point)

Also if we are making mean=0(not subtracting mean from obs) then if I consider this as uncentered mean (For me it is variance around 0) then it is becoming hard to understand how this will lead to uncentered $\text{variance = mean}^2$

Answer 1

Let me try to explain here.

Usually, we calculate the variance by subtracting the mean term and then square it. But here mean (first-moment m_t) is fluctuating like anything at each time "t" and is getting calculated with the influence of past mean as well, also with the influence of beta_1.

So when the 2nd-moment term v_t is getting calculated with a similar approach, when variance is mentioned in the Adam context is understood that we are considering the gradient mean to be equal to zero. This makes sense, since there is no reason to think positive or negative gradients would be more probable than one-another. we are only considering g_t^2 here, so this is called Uncentred variance in the context of calculating 2nd momentum for ADAM.