I have been reading Michael Nielsen’s book online on his website at http://neuralnetworksanddeeplearning.com/chap1.html. I am struggling to understand the second exercise:
When c approaches infinity, wouldn’t make the sigmoid function always output a value close to 1 whereas a perceptron can output 0 or 1.
Let me know if I am missing something or maybe if someone can rephrase the question in a clearer way.
When c approaches infinity, wouldn’t make the sigmoid function always output a value close to 1 whereas a perceptron can output 0 or 1.
This is true only when the original value was positive.
$e^{-cx}$ coverges to $0$ (and $\frac{1}{1 + e^{-cx}}$ converges to $1$) as $c$ approaches infinity only when $x$ is positive. If x is negative, $e^{-cx}$ diverges to infinity and thus $\frac{1}{1 + e^{-cx}}$ converges to $0$.
Thus, the sigmoid function behaves like a step function as $c$ approaches infinity.
