What is numerical stability?

Question

I came across the phrase "numerical stability" several times. But almost in the same context.

I encountered this word mostly in the analytical formula for batch normalization.

$$y = \dfrac{x - \mathbb{E}[x]}{\sqrt{Var[x]+\epsilon}}* \gamma + \beta$$

eps – a value added to the denominator for numerical stability. Default: 1e-5

Is the phenomenon of "numerical instability" happens during the training of neural networks? Or is it a general one in other models also? What is the reason for its occurrence?

Answer 1

You can find a definition for "numerical stability" in mathworld wolframe:

Numerical stability refers to how a malformed input affects the execution of an algorithm. In a numerically stable algorithm, errors in the input lessen in significance as the algorithm executes, having little effect on the final output. On the other hand, in a numerically unstable algorithm, errors in the input cause a considerably larger error in the final output.

In your example, suppose $Var(x)$ is little and truncated to zero in the corresponding computing machine. In that case, you will get INF as a result and it can be problematic to continue the computation.

Therefore, they add a small value such as $\epsilon$ to the division to prevent such a case. As $Var(x) \geqslant 0$, $Var(x) + \epsilon$ will be always greater than zero for any machine with higher precision than the $\epsilon$. So, here, an interpretation of the numerical stability can be like this!