Loss functions are used in training neural networks.

I am interested in knowing the mathematical properties that are necessary for a loss function to participate in gradient descent optimization.

I know some possible candidates that may decide whether a function can be a loss function or not. They include

  1. Continuous at every point in $\mathbb{R}$
  2. Differentiable at every point in $\mathbb{R}$

But, I am not sure whether these two properties are necessary for a function to become a loss function.

Are these two properties necessary? Are there any other mathematical properties that are necessary for a function to become a loss function to participate in gradient descent optimization?

Note that this question is not asking for recommended properties for a loss function. Asking only the mandatory properties in a given context.


Summary: the loss needs to be differentiable, with some caveats.

I will introduce some notation, which I hope is clear: if not I am happy to clarify.

Consider a neural network with parameters $\theta \in \mathbb{R}^d$, which is usually a vector of weights and biases. The gradient descent algorithm seeks to find parameters $\theta_\mathrm{min}$ which minimise the loss function $$\mathcal{L} \colon \mathbb{R}^d \to \mathbb{R}.$$

If this seems abstract, suppose $f(x; \theta)$ is the neural network and $S = \{(x_i, y_i)\}_{i = 1}^n$ is the training set. In binary classification we could have the loss function

$$\mathcal{L}(\theta) = \sum_{i = 1}^n \mathbb{1} \{f(x_i; \theta) \ne y_i\} $$ where $\mathbb{1}$ is the indicator function which is $1$ if the condition is satisfied and zero otherwise. I consider the loss function to be a function of the parameters and not the data, which is fixed.

Gradient descent is performed by the update rule $$ \theta_n \leftarrow \theta_{n - 1} - \gamma \nabla \mathcal{L}(\theta_{n - 1}),$$ yielding new parameters $\theta_n$ which should give a smaller loss $\mathcal{L}(\theta_n)$. The quantity $\gamma$ is the familiar learning rate.

The gradient descent rule requires the gradient $\nabla \mathcal{L}(\theta_{n - 1})$ to be defined, so the loss function must be differentiable. In most texts on calculus or mathematical analysis you'll find the result that if a function is differentiable at a point $x$, it is also continuous at $x$. Obviously there is no hope that we could perform this procedure without knowing the gradient!

In principle, differentiability is sufficient to run gradient descent. That said, unless $\mathcal{L}$ is convex, gradient descent offers no guarantees of convergence to a global minimiser. In practice, neural network loss functions are rarely convex anyway.

I have omitted discussion on stochastic gradient descent, but it does not change the requirements for the loss function. There are alternative techniques such as the proximal gradient method for non-differentiable functions.

An unfortunate technicality I have to mention is that, strictly speaking, if you use the $\mathrm{ReLU}$ activation function, the neural network function $f$ becomes non-differentiable. I discuss this further in this answer. In practice we can assign a value and "pretend" $\mathrm{ReLU}$ is differentiable everywhere.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.