Common loss functions, like the cross-entropy or mean squared error, are chosen because, if you minimize them, you are actually maximizing the likelihood of the parameters given the observed data. In other words, you are trying to find the probability distribution $p(y \mid x; \theta)$, parametrized by $\theta$ (the parameters of your model), which is most consistent with your data $D = \{(x_i, y_i) \}_{i=1}^N$. In the case of binary classification problems, for instance, you assume that $p(y \mid x; \theta)$ is a Bernoulli distribution or a family of Bernoulli distributions (i.e. the likelihood) parametrized by $\theta$ (see more details here). Similarly, in the case of regression problems, you can assume that $p(y \mid x; \theta)$ is a Gaussian. There are other examples of loss functions that are mathematically motivated, for instance, the ELBO or the GAN objective function. The same can be said, for example, for loss functions with regularisation, like the MSE with weight decay. These are mathematically motivated (in this case, for instance, you can derive the MSE with weight decay as a MAP estimate rather than an MLE).
So, loss functions are not chosen arbitrarily, or we don't choose the binary cross-entropy for binary classification problems just because everyone else does (well, ideally, this shouldn't be the reason, but I guess many people don't know the details that I've just mentioned above). We often model our problem with probability theory and we try to find a probability distribution (or the mean/mode or even variance of such a probability distribution) that most likely generated our data.
Now, if you train a neural network with gradient descent and backpropagation, it's assumed that you can also find the derivative of your loss function with respect to the parameters of your model. So, this is one of the properties that a loss function is expected to have in this training regime, i.e. it needs to be differentiable.
One of the most commonly used metrics to assess the quality of a classifier is the accuracy, which I think is not differentiable (although I don't have the proof and I didn't think too much about it). So, that's probably one of the reasons why people wouldn't use the accuracy as a loss function anyway. Now, keep in mind that the accuracy can actually be a very bad metric to assess the quality of your model because, if your dataset is unbalanced (and you didn't check that and the true distribution is not), your classifier could simply always predict the most common class and get a high accuracy, but this would be misleading.
So, in summary, a function is often used as a loss function (in the context of training neural networks with GD) if
- it is differentiable, and,
- if you minimize it, you're actually trying to learn a probability distribution
Note that the second property may not always be a requirement, but all the loss functions that I can think of right now have these two properties.
On the other hand, a function is used as a metric if you care about what that specific function evaluates. For example, if you use the accuracy, you want to know, on average, how many mistakes the model makes, but you don't care whether it makes more mistakes for one class than the other. However, you can also use other functions, like precision and recall, which evaluate other behaviors, like how many mistakes you make for one class rather than the other. This can make sense, for example, in the context of detecting a disease.
In principle, you could use a loss function also as a metric. However, is that really what you want? For instance, if you used the MSE as a metric, it's not clear how small it must be in order to have a "good" model. It all depends on which behavior of the model that you want to evaluate. You can evaluate different behaviors of the model, so that's why you often have more than one metric.
