Loss functions are used in training neural networks.

I am interested in knowing the mathematical properties that are necessary for a loss function to participate in gradient descent optimization.

I know some possible candidates that may decide whether a function can be a loss function or not. They include

  1. Continuous at every point in $\mathbb{R}$
  2. Differentiable at every point in $\mathbb{R}$

But, I am not sure whether these two properties are necessary for a function to become a loss function.

Are these two properties necessary? Are there any other mathematical properties that are necessary for a function to become a loss function to participate in gradient descent optimization?

Note that this question is not asking for recommended properties for a loss function. Asking only the mandatory properties in a given context.


1 Answer 1


Summary: the loss needs to be differentiable, with some caveats.

I will introduce some notation, which I hope is clear: if not I am happy to clarify.

Consider a neural network with parameters $\theta \in \mathbb{R}^d$, which is usually a vector of weights and biases. The gradient descent algorithm seeks to find parameters $\theta_\mathrm{min}$ which minimise the loss function $$\mathcal{L} \colon \mathbb{R}^d \to \mathbb{R}.$$

If this seems abstract, suppose $f(x; \theta)$ is the neural network and $S = \{(x_i, y_i)\}_{i = 1}^n$ is the training set. In binary classification we could have the loss function

$$\mathcal{L}(\theta) = \sum_{i = 1}^n \mathbb{1} \{f(x_i; \theta) \ne y_i\} $$ where $\mathbb{1}$ is the indicator function which is $1$ if the condition is satisfied and zero otherwise. I consider the loss function to be a function of the parameters and not the data, which is fixed.

Gradient descent is performed by the update rule $$ \theta_n \leftarrow \theta_{n - 1} - \gamma \nabla \mathcal{L}(\theta_{n - 1}),$$ yielding new parameters $\theta_n$ which should give a smaller loss $\mathcal{L}(\theta_n)$. The quantity $\gamma$ is the familiar learning rate.

The gradient descent rule requires the gradient $\nabla \mathcal{L}(\theta_{n - 1})$ to be defined, so the loss function must be differentiable. In most texts on calculus or mathematical analysis you'll find the result that if a function is differentiable at a point $x$, it is also continuous at $x$. Obviously there is no hope that we could perform this procedure without knowing the gradient!

In principle, differentiability is sufficient to run gradient descent. That said, unless $\mathcal{L}$ is convex, gradient descent offers no guarantees of convergence to a global minimiser. In practice, neural network loss functions are rarely convex anyway.

I have omitted discussion on stochastic gradient descent, but it does not change the requirements for the loss function. There are alternative techniques such as the proximal gradient method for non-differentiable functions.

An unfortunate technicality I have to mention is that, strictly speaking, if you use the $\mathrm{ReLU}$ activation function, the neural network function $f$ becomes non-differentiable. I discuss this further in this answer. In practice we can assign a value and "pretend" $\mathrm{ReLU}$ is differentiable everywhere.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .