How to tackle the human error made in labeling datasets for classification tasks like facial expression recognition?

Question

I am working on the Facial Expression Recognition Task. One of the most challenging tasks that I faced was human error in labeling the datasets (ex: let's say FER2013).

Are there anyways to Handle incorrect labeling of datasets in the classification tasks? Will using clustering methods and treating it as an unsupervised Learning problem (just a thought not have a clear idea) solve this kind of issue? or are any other methods available without treating it as an unsupervised task?

Answer 1

In general the only way to deal with this is by quantifying these labeling mistakes in the output of the model, since the model will learn for them. And in many cases these are not really mistakes, but they encode the ambiguity of the task, particularly in interpreting facial emotions.

For this you need to use a model that can estimate Aleatoric uncertainty (this is data uncertainty). For classification you can use the cross-entropy loss combined with the sampling softmax function, which replaces a standard Dense layer with one that computes a Gaussian distribution for the softmax logits (so it predicts mean logit and variance logits).

This method is described in the paper "What uncertainties do we need in bayesian deep learning for computer vision?" (link), where the authors can also disentangle aleatoric from epistemic uncertainty.

I have run experiments exactly on the facial emotion recognition task (using the FER+ dataset), which you can find in my paper "A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement" (link). In this paper I show multiple uncertainty quantification methods and their ability to produce disentangled probabilities. If you only need aleatoric uncertainty, then only the sampling softmax is needed.

I even have an implementation of this layer in my keras-uncertainty library.