# What are the necessary mathematical properties to be a loss function in gradient based optimizations?

Loss functions are used in training neural networks.

I am interested in knowing the mathematical properties that are necessary for a loss function to participate in gradient descent optimization.

I know some possible candidates that may decide whether a function can be a loss function or not. They include

1. Continuous at every point in $$\mathbb{R}$$
2. Differentiable at every point in $$\mathbb{R}$$

But, I am not sure whether these two properties are necessary for a function to become a loss function.

Are these two properties necessary? Are there any other mathematical properties that are necessary for a function to become a loss function to participate in gradient descent optimization?

Note that this question is not asking for recommended properties for a loss function. Asking only the mandatory properties in a given context.

Summary: the loss needs to be differentiable, with some caveats.

I will introduce some notation, which I hope is clear: if not I am happy to clarify.

Consider a neural network with parameters $$\theta \in \mathbb{R}^d$$, which is usually a vector of weights and biases. The gradient descent algorithm seeks to find parameters $$\theta_\mathrm{min}$$ which minimise the loss function $$\mathcal{L} \colon \mathbb{R}^d \to \mathbb{R}.$$

If this seems abstract, suppose $$f(x; \theta)$$ is the neural network and $$S = \{(x_i, y_i)\}_{i = 1}^n$$ is the training set. In binary classification we could have the loss function

$$\mathcal{L}(\theta) = \sum_{i = 1}^n \mathbb{1} \{f(x_i; \theta) \ne y_i\}$$ where $$\mathbb{1}$$ is the indicator function which is $$1$$ if the condition is satisfied and zero otherwise. I consider the loss function to be a function of the parameters and not the data, which is fixed.

Gradient descent is performed by the update rule $$\theta_n \leftarrow \theta_{n - 1} - \gamma \nabla \mathcal{L}(\theta_{n - 1}),$$ yielding new parameters $$\theta_n$$ which should give a smaller loss $$\mathcal{L}(\theta_n)$$. The quantity $$\gamma$$ is the familiar learning rate.

The gradient descent rule requires the gradient $$\nabla \mathcal{L}(\theta_{n - 1})$$ to be defined, so the loss function must be differentiable. In most texts on calculus or mathematical analysis you'll find the result that if a function is differentiable at a point $$x$$, it is also continuous at $$x$$. Obviously there is no hope that we could perform this procedure without knowing the gradient!

In principle, differentiability is sufficient to run gradient descent. That said, unless $$\mathcal{L}$$ is convex, gradient descent offers no guarantees of convergence to a global minimiser. In practice, neural network loss functions are rarely convex anyway.

I have omitted discussion on stochastic gradient descent, but it does not change the requirements for the loss function. There are alternative techniques such as the proximal gradient method for non-differentiable functions.

An unfortunate technicality I have to mention is that, strictly speaking, if you use the $$\mathrm{ReLU}$$ activation function, the neural network function $$f$$ becomes non-differentiable. I discuss this further in this answer. In practice we can assign a value and "pretend" $$\mathrm{ReLU}$$ is differentiable everywhere.