1
$\begingroup$

I came across the hinge loss function for training a neural network model, but I did not know the analytical form for the same.

I can write the mean squared error loss function (which is more often used for regression) as

$$\sum\limits_{i=1}^{N}(y_i - \hat{y_i})^2$$

where $y_i$ is the desired output in the dataset, $\hat{y_i}$ is the actual output by the model, and $N$ is the total number of instances in our dataset.

Similarly, what is the (basic) expression for hinge loss function?

$\endgroup$
1
  • $\begingroup$ The idea behind hinge loss (not obvious from its expression) is that the NN must predict with confidence i.e.its prediction score must exceed a certain threshold (a hyperparameter) for the loss to be 0. Hence while training the NN tries to predict with maximum confidence or exceed the threshold so that loss is 0. $\endgroup$ – user9947 Feb 11 at 15:57
1
$\begingroup$

The hinge loss/error function is the typical loss function used for binary classification (but it can also be extended to multi-class classification) in the context of support vector machines, although it can also be used in the context of neural networks, as described here.

The hinge loss function is defined as follows

$$ \ell(y) = \max(0, 1-t \cdot y) \tag{1}\label{1}, $$ where

  • $t = \{-1, 1\}$ is the label (so, if your labels are in the set $\{0, 1 \}$, you will have to first map them to $\{-1, 1\}$)
  • $y$ is the output of the classifier (e.g. in the context of the linear SVM, $y=\mathbf{w} \cdot \mathbf{x}+b$, where $\mathbf{w}$ and $b$ are the parameter of the hyper-plane)

This means that the loss in equation \ref{1} is always non-negative. If you're familiar with the ReLU, this loss should look familiar to you. In fact, their plots are very similar.

For more details, you probably should start with the related Wikipedia article, then maybe one of the many machine learning books that covers support vector machines, for example, Pattern Recognition and Machine Learning (2006) by Christopher Bishop, chapter 7 (page 325).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.