An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying?

Question

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following:

The classical activation functions that were used early in the development of neural networks were the sign, sigmoid, and the hyperbolic tangent functions: $$\Phi(v) = \text{sign}(v) \ \ \text{(sign function)} \\ \Phi(v) = \dfrac{1}{1 + e^{-v}} \ \ \text{(sigmoid function)} \\ \Phi(v) = \dfrac{e^{2v} - 1}{e^{2v} + 1} \ \ \text{(tanh function)}$$ While the sign activation can be used to map to binary outputs at prediction time, its non-differentiability prevents its use for creating the loss function at training time. For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation.

I am having trouble understanding this part:

While the sign activation can be used to map to binary outputs at prediction time, its non-differentiability prevents its use for creating the loss function at training time. For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation.

I've read over this a number of times, but I still don't have a good idea of what it is saying (or at least the point it is trying to make). What is this actually saying? What is the point this is trying to make? Perhaps a more detailed explanation of what this is saying will clarify it for me.

Answer 1

sign is not continuous and not differentiable. Let's say it is defined as follows:

$$ \text{sing}(a) = \begin{cases} +1 & \text{if $a>0$}\\ -1 & \text{otherwise} \end{cases} $$

where $a = \textbf{w}^T\textbf{x}$ is a linear pre-activation function.

The graph will look as follows:

It is not continuous and, therefore, is not differentiable everywhere. Even if we take the derivative of such a function, it will be equal to zero, since the function is flat. That is, sign does not tell us how good our predictions are, it only says whether they were correct or not.

More formally, it does not return a gradient that represents the direction in which we should update the $a$ wrt $\textbf{w}$. Instead, we can only move the line (in 2d) trying to correctly classify as many data points as possible by manually changing each weight. If $a$ were non-linear, it would be impossible to fit a non-linear function by manually changing the weights, since changes in an individual weight would lead to unpredictable changes in the function.