How to classify when the label is seldom known

Question

When I think about classification I think of the cancer/not cancer example. You have a bunch of attributes and you know whether the person had cancer during the relevant time period and you determine which attributes predict that result.

I work in a highly-regulated industry that serves the public. There are certain people we are not allowed to do business with, let's say because they will use our service for illegal purposes. Sometimes we tell the (potential) customer "yes" and sometimes "no".

When we say "yes" and the potential customer intends to use our service for illegal activity, they certainly won't inform us of the mistake.

Likewise, when we say "no" the potential customer will sometimes go away, will sometimes complain, but that potential customer will not self-identify and say "yes, you are correct, I intended to use your service for illegal activity".

Occasionally we will receive a report from a 3rd-party that will label a customer, but these reports are a tiny fraction of the number of customers. Unlike the cancer classification we almost always don't know the actual label, we only know what we guessed.

What techniques should we consider to measure our accuracy?

Answer 1

Domain knowledge and cluster interpretation

When you have no (or very limited) gold standard labels for your dataset, traditional performance metrics that require knowing how often you're correct (like accuracy, sensitivity, or specificity) simply won't work. If this is the case, you'll need to examine your outliers to see if they make sense in the context of your particular field. You should look at what features are responsible for a sample being classified as an anomaly, for example, you might see that agents classified as spam send a huge number of emails per hour, or that a suspected money laundering bank account interacts frequently with many offshore accounts. In most cases, you will never know with absolute certainty that your classification is correct, but if you can justify the classification in terms of what is known about your particular domain, it can lend credence to a model.

Another approach is to perform unsupervised clustering on your dataset. Hopefully, you should see that your method identifies anomalies which are somehow different from other datapoints, appearing as outliers or a distinct group of samples unlike the others. You will need to do more investigation to be confident that the clusters represent normal people and fraudsters, but clustering can help you identify that there are indeed some samples that appear markedly different from the others. If you can show that certain samples show similar hallmarks of illegal activity, it also helps to gain confidence in what your model is doing.