When I think about classification I think of the cancer/not cancer example. You have a bunch of attributes and you know whether the person had cancer during the relevant time period and you determine which attributes predict that result.
I work in a highly-regulated industry that serves the public. There are certain people we are not allowed to do business with, let's say because they will use our service for illegal purposes. Sometimes we tell the (potential) customer "yes" and sometimes "no".
When we say "yes" and the potential customer intends to use our service for illegal activity, they certainly won't inform us of the mistake.
Likewise, when we say "no" the potential customer will sometimes go away, will sometimes complain, but that potential customer will not self-identify and say "yes, you are correct, I intended to use your service for illegal activity".
Occasionally we will receive a report from a 3rd-party that will label a customer, but these reports are a tiny fraction of the number of customers. Unlike the cancer classification we almost always don't know the actual label, we only know what we guessed.
What techniques should we consider to measure our accuracy?