0
$\begingroup$

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following:

The classical activation functions that were used early in the development of neural networks were the sign, sigmoid, and the hyperbolic tangent functions: $$\Phi(v) = \text{sign}(v) \ \ \text{(sign function)} \\ \Phi(v) = \dfrac{1}{1 + e^{-v}} \ \ \text{(sigmoid function)} \\ \Phi(v) = \dfrac{e^{2v} - 1}{e^{2v} + 1} \ \ \text{(tanh function)}$$ While the sign activation can be used to map to binary outputs at prediction time, its non-differentiability prevents its use for creating the loss function at training time. For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation.

I am having trouble understanding this part:

While the sign activation can be used to map to binary outputs at prediction time, its non-differentiability prevents its use for creating the loss function at training time. For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation.

I've read over this a number of times, but I still don't have a good idea of what it is saying (or at least the point it is trying to make). What is this actually saying? What is the point this is trying to make? Perhaps a more detailed explanation of what this is saying will clarify it for me.

$\endgroup$
6
  • $\begingroup$ sign is not continuous and not differentiable, so we cannot get a derivative of it and hence apply gradient descent optimization. Have you looked at this answer. Does it answer your question? $\endgroup$ Jun 25 at 11:37
  • $\begingroup$ @ArayKarjauv Your answer partially clarifies things, but it doesn't explain how this all relates to the perceptron criterion. $\endgroup$ Jun 25 at 11:38
  • $\begingroup$ Since the perceptron becomes nondifferentiable, we cannot compute the error and propagate it back through all the nodes. Instead, there are iterative algorithms to optimize the perceptron (see the last part of the answer). $\endgroup$ Jun 25 at 11:45
  • $\begingroup$ @ArayKarjauv Eh, I don't really see how this addresses the part about the perceptron criterion. $\endgroup$ Jun 25 at 11:54
  • $\begingroup$ What do you mean by the "perceptron criterion"? $\endgroup$ Jun 25 at 11:57
2
$\begingroup$

sign is not continuous and not differentiable. Let's say it is defined as follows:

$$ \text{sing}(a) = \begin{cases} +1 & \text{if $a>0$}\\ -1 & \text{otherwise} \end{cases} $$

where $a = \textbf{w}^T\textbf{x}$ is a linear pre-activation function.

The graph will look as follows:

enter image description here

It is not continuous and, therefore, is not differentiable everywhere. Even if we take the derivative of such a function, it will be equal to zero, since the function is flat. That is, sign does not tell us how good our predictions are, it only says whether they were correct or not.

More formally, it does not return a gradient that represents the direction in which we should update the $a$ wrt $\textbf{w}$. Instead, we can only move the line (in 2d) trying to correctly classify as many data points as possible by manually changing each weight. If $a$ were non-linear, it would be impossible to fit a non-linear function by manually changing the weights, since changes in an individual weight would lead to unpredictable changes in the function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.