# What is numerical stability?

I came across the phrase "numerical stability" several times. But almost in the same context.

I encountered this word mostly in the analytical formula for batch normalization.

$$y = \dfrac{x - \mathbb{E}[x]}{\sqrt{Var[x]+\epsilon}}* \gamma + \beta$$

eps – a value added to the denominator for numerical stability. Default: 1e-5

Is the phenomenon of "numerical instability" happens during the training of neural networks? Or is it a general one in other models also? What is the reason for its occurrence?

In your example, suppose $$Var(x)$$ is little and truncated to zero in the corresponding computing machine. In that case, you will get INF as a result and it can be problematic to continue the computation.
Therefore, they add a small value such as $$\epsilon$$ to the division to prevent such a case. As $$Var(x) \geqslant 0$$, $$Var(x) + \epsilon$$ will be always greater than zero for any machine with higher precision than the $$\epsilon$$. So, here, an interpretation of the numerical stability can be like this!