How to use strong labels in image classification?

Question

I have a dataset where I have the labels cancer & non-cancer, and I also have localized pixel-level annotation masks of important regions/features in the images.

In a binary classification task, how do I optimize my deep learning model to focus on the annotated regions of the image and ignore or pay less attention to the regions outside the annotations?

Additionally, I have a set of images with only image level labels. Is there a way to use both localized labels (strong labels) and image level labels (weak labels) in the same model?

Answer 1

This is a great question. Sorry I only have 1 reputation point, so I can't upvote. The optimal solution likely has to be empirically determined. I don't think it can be determined a priori. It would also depend on how you define as optimal (high sensitivity, high specificity, high AUC, or high accuracy).

One approach would be to develop a segmentation model and any image with "tumor pixels" would be classified at the image level as "tumor". I don't know for sure (and no one really knows for sure), but I think this method would yield higher sensitivity (but possibly lower specificity) than a straight image classifier. Which one is likely to be better depends on a lot of things, including how much training data you have, the quality of annotations, and the quality of images.

To answer your second question, yes there is. There are probably several ways to integrate the different levels of information. One simple way is to develop a model with weak labels and a model with strong labels and then do a probability fusion, similar to how it is done for multimodal data (see article).