Aside from dice score, what other good metrics are used to evaluate segmentation models?

Question

I have a segmentation which outputs only one channel image (2 class segmentation). I have used dice score for most of the time, but now higher powers in my team want me to expand evaluation metrics for segmentation model (if it's even possible). I have done some research and as far as right now I have found mainly that everybody uses dice score, and sometimes pixel to pixel binary accuracy, but for the latter seems not the best idea.

If anybody knows something exciting or useful, I'd be glad to hear from them.

Answer 1

Typical metrics used with segmentation problems are Recall, Precision and the F1 Score (similar or the same as the Dice score depending on the definition used). These can be evaluated per class or for all classes together, commonly referred to as micro and macro averages.

Taking it further, you may wish to have a metric more robust to changes in the threshold. Here the Area under the Curve (AUC) metric is commonly used.

For a more sophisticated analysis you may also be interested in perceptual losses. These quantify how similar an image looks as perceived by people. This is particularly useful if say the shape of the prediction is important but small shifts or scaling does not matter. Have a look at SSIM and LPIPS losses for more information on these.

TorchMetrics may be a good place to look implementations and available metrics.

Answer 2

I agree fully with @a crazy Minon's answer. I will just slightly expand on it and provide a couple of additional references.

While Dice is a popular metric for evaluating segmentation, it is certainly not the only one. You are right in thinking that pixel accuracy is a poor choice of evaluation metric. The main issue is that it performs poorly when when there is class imbalance, which is often the case in imaging data.

I will add that Intersection over Unions (IoU) is another metric that is frequently employed to evaluate segmentation performance. It is also known as the Jaccard Index. The articles "Metrics to Evaluate your Semantic Segmentation Model" and "All the segmentation metrics!" provide good simplified introductions to various commonly used segmentation metrics. While Dice and IoU are similar and are positively correlated, they are not equivalent, as explained by this StackOverflow answer. New metrics are also being developed--such as the Boundary Jaccard--to overcome limitations of current metrics, and comparisons of these metrics have been published for specific applications (see example ref, which lists 33 evaluation metrics for segmentation in Table 1).

Finally, if your interest is really for one class, then accuracy, sensitivity, and specificity for segmenting that class alone can be useful metrics.

The powers that be at your institution are wise in asking for multiple evaluation metrics because each metric has its limitations and no single metric can fully capture the performance of a segmentation network.

Answer 3

https://metrics-reloaded.dkfz.de/ -> Link to cite.

This is an excellent site to see how to gauge which type of task and metric to use.

Since you specifically want segmentation, just opt for that and have a look!

Reference: https://arxiv.org/abs/2206.01653