0
$\begingroup$

In section 3 paragraph 2 of Batch Normalization: Accelerating Deep Network Training b y Reducing Internal Covariate Shift paper (https://arxiv.org/abs/1502.03167) they say that normalizing a layer's input may change what it represents, I understand this. But what do they mean in the bolded part?

For instance, normalizing the inputs of a sigmoid would constrain them to the linear regime of the nonlinearity.

$\endgroup$

1 Answer 1

2
$\begingroup$

The normalization makes the signal small enough to remain in the region of the sigmoid that can be well approximated by a straight line.

The idea is the same as in electronics: https://en.wikipedia.org/wiki/Small-signal_model

$\endgroup$
2
  • $\begingroup$ Oh so near the range of around [-1, 1] on the horizontal axis where the slope is close to being a straight line? $\endgroup$
    – rkuang25
    Dec 29, 2022 at 4:45
  • $\begingroup$ Yes, that's it. The sigmoid is also almost linear, flat, for large absolute values. The hard sigmoid is a piecewise linear approximation to the sigmoid based on these three regions: large negative, around zero, large positive. $\endgroup$ Dec 29, 2022 at 6:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .