Summary: the loss needs to be differentiable, with some caveats.
I will introduce some notation, which I hope is clear: if not I am happy to clarify.
Consider a neural network with parameters $\theta \in \mathbb{R}^d$, which is usually a vector of weights and biases. The gradient descent algorithm seeks to find parameters $\theta_\mathrm{min}$ which minimise the loss
function
$$\mathcal{L} \colon \mathbb{R}^d \to \mathbb{R}.$$
If this seems abstract, suppose $f(x; \theta)$ is the neural network and $S = \{(x_i, y_i)\}_{i = 1}^n$ is the training set. In binary classification we could have the loss function
$$\mathcal{L}(\theta) = \sum_{i = 1}^n \mathbb{1} \{f(x_i; \theta) \ne y_i\} $$
where $\mathbb{1}$ is the indicator function which is $1$ if the condition is satisfied and zero otherwise. I consider the loss function to be a function of
the parameters and not the data, which is fixed.
Gradient descent is performed by the update rule
$$ \theta_n \leftarrow \theta_{n - 1} - \gamma \nabla \mathcal{L}(\theta_{n - 1}),$$
yielding new parameters $\theta_n$ which should give a smaller loss $\mathcal{L}(\theta_n)$. The quantity $\gamma$ is the familiar learning rate.
The gradient descent rule requires the gradient $\nabla \mathcal{L}(\theta_{n - 1})$ to be defined, so the loss function must be differentiable. In most texts on calculus or mathematical analysis you'll find the result that if a function is
differentiable at a point $x$, it is also continuous at $x$. Obviously there is no
hope that we could perform this procedure without knowing the gradient!
In principle, differentiability is sufficient to run gradient descent. That said, unless $\mathcal{L}$ is convex, gradient descent offers no guarantees of convergence to a global minimiser. In practice, neural network loss functions are rarely convex
anyway.
I have omitted discussion on stochastic gradient descent, but it does not change the requirements for the loss function. There are alternative techniques such as the proximal gradient method for non-differentiable functions.
An unfortunate technicality I have to mention is
that, strictly speaking, if you use the $\mathrm{ReLU}$ activation function, the
neural network function $f$ becomes non-differentiable. I discuss this further in this answer. In practice we can assign
a value and "pretend" $\mathrm{ReLU}$ is differentiable everywhere.
