I'm aware of metrics like accuracy (correct predictions / total predictions) for models that classify things. However, I'm working on a model that outputs the probability of a datapoint belonging to one of two classes. What metrics can/should be used to evaluate these types of models?

I'm currently using mean squared error, but I would like to know if there are other metrics, and what the advantages/disadvantages of those metrics are.


For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within some ratio of the true value. So an estimate of $\hat{p} = 0.1$ is a better estimate if the true value is $p = 0.2$ than if the true value is $p = 0.01$ (even though the latter value is closer and would score better under MSE). It also applies that you care that 0.9 is by the same logic "closer" to 0.8 than it is to 0.97. With cross-entropy loss, extreme confidence (predicting close to $0$ or close to $1$) is penalised more heavily when it is wrong.

For completeness, the loss function (per data point) is:

$$\mathcal{L}(\hat{y},y) = -(y\text{log}(\hat{y})+ (1-y)\text{log}(1-\hat{y}))$$

This is likely to be the same loss function as you are using for your objective (or at least it should be), so for tests, simply also use it as your metric*.

$\hat{y}$ is your predicted probability of being in class A, and can be in range $[0,1]$. Ideally you have ground truth probabilities and $y$ is also in that range. In which case the only problem is that the "perfect" score is no longer $0$ but some positive number. If that bothers you, then you could offset by the perfect score, pre-calculating it on each data set (just set $\hat{y} = y$ for each item and you will find the minimum possible score).

If you don't have ground truth probabilities, but you do have classes, then $y$ will either be 0 or 1, and the metric still works. To get an accurate metric, you will need enough samples that the relative frequencies of each class depending on input has a significant effect. That is, you need more data, both training and test, in order to train for accurate probabilities instead of targeting simpler classification accuracy metrics.

Similar logic also works for multi-class probabilities. However, many off-the-shelf libraries use an optimisation in the loss function - assuming only one true class - which makes using probabilities as ground truth impossible. You might therefore need to write your own loss function and gradient functions based on multi-class cross-entropy loss in that case.

* I am making the assumption here that you use standard conventions for noting loss functions (typically per item), cost functions (typically aggregated across a data set and possibly multiple loss functions) and metric functions which don't have to be differentiable or usable as either of the former. The cost function is usually also fed into optimisers as the objective function - i.e. it has the important job of driving parameters to reach a maximum or minimum value. For gradient based solvers, that means it must be differentiable.

  • 1
    $\begingroup$ Thank you! This is helpful. $\endgroup$ – Shylock Jul 5 '19 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.