Imagine that I am training a model to classify handwritten digits. Suppose there are some bad quality images that could be classified by a human as either 0 or 8, 1 or 7 or other commonly misclassified pair of digits. My question is, should I simply remove such ambiguous samples? Should I annotate it as the most similar digit, even though there are other similar answers? Should I present it repeat the sample, presenting it once per each 'acceptable' answer?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This depends on the behaviour you want. If the ambiguous sample's ground truth is classified by a range of people, your network will get an average* based on that group. If it's only by one person, your network will be biased to how that one person classifies these samples.
Alternatively, depending on your loss function, you could train the network to classify ambiguous samples with an ambiguous label. For example, if 40 people label the digit as 1, and 60 as 7, you could have the desired output for the network be 0.4 for 1 and 0.6 for 7 (assuming it's a probability).
This doesn't have an exact answer, it's whatever behaviour you deem best for your scenario. If you want to keep it simple, you can remove the samples and then see how the network performs on these ambiguous samples at testing time. Assuming your dataset is good, you'll probably find the network performs about the same on these samples anyway.
