As per these slides on page 35:

Sigmoids saturate and kill gradients.

when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero.

the gradient and almost no signal will flow through the neuron to its weights and recursively to its data.

So, if the gradient is close to zero, then the error correction would be very minimal. But why would that cause that no signal flow through the neuron?

$$w(n+1) = w(n) - \text{gradient}$$

That would only cause the weights not to change.

  • 3
    $\begingroup$ I think they mean no signal will flow through the backpropagation since one node close to zero leads to turn off all the gradient of the previous nodes. $\endgroup$ – CuCaRot Feb 1 at 7:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.