# What does linear regime of nonlinearity mean in normalisation?

In section 3 paragraph 2 of Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift paper (https://arxiv.org/abs/1502.03167) they say that normalizing a layer's input may change what it represents, I understand this. But what do they mean in the bolded part?

For instance, normalizing the inputs of a sigmoid would constrain them to the linear regime of the nonlinearity.

The normalization makes the signal small enough to remain in the region of the sigmoid that can be well approximated by a straight line.

The idea is the same as in electronics: https://en.wikipedia.org/wiki/Small-signal_model

• Oh so near the range of around [-1, 1] on the horizontal axis where the slope is close to being a straight line? Commented Dec 29, 2022 at 4:45
• Yes, that's it. The sigmoid is also almost linear, flat, for large absolute values. The hard sigmoid is a piecewise linear approximation to the sigmoid based on these three regions: large negative, around zero, large positive. Commented Dec 29, 2022 at 6:30