When is it appropriate to use information like sex or race in ethical machine learning?

Question

I'm a little confused on best practices regarding ethical ML. Specifically, I've seen in some courses that when building a model that affects people, it's helpful to have sensitive personal information related to sex, race, etc., to help ensure that the model distributes its error evenly across populations and avoids disparities.

I've also seen (in practice on the job and in recommendations) that this kind of information should be entirely omitted from your data to ensure the "blindness" of the model to protected and sensitive classes.

These directives seem at odds with each other. Has any kind of rubric been developed to help modelers determine when it is and is not appropriate to use this kind of information in a model?

Answer 1

Those suggestions are actually not in contradiction with each other.

Having data for testing and using them for training are two completely different things. Let's take the infamous example of Amazon automatic recruiting algorithm. The model was trained on real people curricula tp rank them and suggest this way the best candidates to hire. After deploying it it became clear pretty soon that the model was gender biased, cause it was not ranking female curricula high even when all other requirements were met.

Why was the model biased? Because humans are also biased and we are those who labelled the data from which the model learn. So if a human recruiter compare the same curricula with a male name and a female name chances are it will select the curriculum with a male name.

So far so good, we spotted a potential cause for the bias, so we can just retrain the model without the gender variable and it should be fine. But can we trust the model will not be gender biased anymore? Well not 100%, there are many correlations between gender and other variables, and the model might still learn a biased behavior by "learning" the gender variable from other variables.

So that's why we want to keep the gender variable in our data, but only to use it in test phase, not as an extra feature to feed to the model (this time it was trained without) but as a variable to use to compute statistics out of the model predictions, i.e. how many males and females does the model choose after being trained without information about gender?