# How should we interpret this figure that relates the perceptron criterion and the hinge loss?

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.2 Relationship with Support Vector Machines says the following:

The perceptron criterion is a shifted version of the hinge-loss used in support vector machines (see Chapter 2). The hinge loss looks even more similar to the zero-one loss criterion of Equation 1.7, and is defined as follows: $$L_i^{svm} = \max\{ 1 - y_i(\overline{W} \cdot \overline{X}_i), 0 \} \tag{1.9}$$ Note that the perceptron does not keep the constant term of $$1$$ on the right-hand side of Equation 1.7, whereas the hinge loss keeps this constant within the maximization function. This change does not affect the algebraic expression for the gradient, but it does change which points are lossless and should not cause an update. The relationship between the perceptron criterion and the hinge loss is shown in Figure 1.6. This similarity becomes particularly evident when the perceptron updates of Equation 1.6 are rewritten as follows: $$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{(\overline{X}, y) \in S^+} y \overline{X} \tag{1.10}$$ Here, $$S^+$$ is defined as the set of all misclassified training points $$\overline{X} \in S$$ that satisfy the condition $$y(\overline{W} \cdot \overline{X}) < 0$$. This update seems to look somewhat different from the perceptron, because the perceptron uses the error $$E(\overline{X})$$ for the update, which is replaced with $$y$$ in the update above. A key point is that the (integer) error value $$E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$$ can never be $$0$$ for misclassified points in $$S^+$$. Therefore, we have $$E(\overline{X}) = 2y$$ for misclassified points, and $$E(X)$$ can be replaced with $$y$$ in the updates after absorbing the factor of $$2$$ within the learning rate.

Equation 1.6 is as follows:

$$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{\overline{X} \in S} E(\overline{X})\overline{X}, \tag{1.6}$$ where $$S$$ is a randomly chosen subset of training points, $$\overline{X} = [x_1, \dots, x_d]$$ is a data instance (vector of $$d$$ feature variables), $$\overline{W} = [w_1, \dots, w_d]$$ are the weights, $$\alpha$$ is the learning rate, and $$E(\overline{X}) = (y - \hat{y})$$ is an error value, where $$\hat{y} = \text{sign}\{ \overline{W} \cdot \overline{X} \}$$ is the prediction and $$y$$ is the observed value of the binary class variable.

Equation 1.7 is as follows:

$$L_i^{(0/1)} = \dfrac{1}{2} (y_i - \text{sign}\{ \overline{W} \cdot \overline{X_i} \})^2 = 1 - y_i \cdot \text{sign} \{ \overline{W} \cdot \overline{X_i} \} \tag{1.7}$$

And figure 1.6 is as follows:

Figure 1.6 looks unclear to me. What is figure 1.6 showing, and how is it relevant to the point that the author is trying to make?

• Hi! you didn't accept my answer. Is it because my answer was unclear? Please feel free to clarify, I was a bit busy when I wrote the answer, but now I can answer any doubts you have about the nswer – DuttaA Feb 4 at 10:41
• I was also confused about this specific part of the book and came accross the following video explaining hinge loss. Hope it might help: youtube.com/watch?v=PM2MSAYmzXM – user45643 Mar 24 at 1:22