Why is the perceptron criterion function differentiable?

Question

I'm reading chapter one of the book called Neural Networks and Deep Learning from Aggarwal.

In section 1.2.1.1 of the book, I'm learning about the perceptron. One thing that book says is, if we use the sign function for the following loss function: $\sum_{i=0}^{N}[y_i - \text{sign}(W * X_i)]^2$, that loss function will NOT be differentiable. Therefore, the book suggests us to use, instead of the sign function in the loss function, the perceptron criterion which will be defined as:

$$ L_i = \max(-y_i(W * X_i), 0) $$

The question is: Why is the perceptron criterion function differentiable? Won't we face a discontinuity at zero? Is there anything that I'm missing here?

Answer 1

$\max(-y_i(w x_i), 0)$ is not partial derivable respect $w$ if $w x_i=0$.

Loss functions are problematic when not derivable in some point, but even more when they are flat (constant) in some interval of the weights.

Assume $y_i = 1$ and $w x_i < 0$ (that is, an error of type "false negative").

In this case, function $[y_i - \text{sign}(w x_i)]^2 = 4$. Derivative on all interval $w x_i < 0$ is zero, thus, the learning algorithm has no any way to decide if it is better increase or decrease $w$.

In same case, $\max(-y_i(w x_i), 0) = - w x_i$, partial derivative is $-x_i$. The learning algorithm knows that it must increase $w$ value if $x_i>0$, decrease otherwise. This is the real reason this loss function is considered more practical than previous one.

How to solve the problem at $w x_i = 0$ ? simply, if you increase $w$ and the result is an exact $0$, assign to it a very small value, $w=\epsilon$. Similar logic for remainder cases.

Answer 2

Since we're dealing with real-values variables, it is almost certainly the case that the argument of the function will not be $0$.

If you care strongly about that point, you can just use sub-gradients instead (and we do have sub-gradients for this function, so there is no problem).