4
$\begingroup$

As per these slides on page 35:

Sigmoids saturate and kill gradients.

when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero.

the gradient and almost no signal will flow through the neuron to its weights and recursively to its data.

So, if the gradient is close to zero, then the error correction would be very minimal. But why would that cause that no signal flow through the neuron?

$$w(n+1) = w(n) - \text{gradient}$$

That would only cause the weights not to change.

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ I think they mean no signal will flow through the backpropagation since one node close to zero leads to turn off all the gradient of the previous nodes. $\endgroup$
    – CuCaRot
    Feb 1, 2021 at 7:47

0

Reset to default

You must log in to answer this question.