Should I define my problem as image segmentation or detection?

Question

I have a problem and have to decide wether it's an object detection or object segmentation problem. I want to use Yolov8 for training. We already have hundrets of images but they aren't labeled yet. So before we start labeling we have to clearfy what we want. Maybe you can help me. I will provide the facts:

• 8 classes of objects
• every class occurs 0 to n times per image
• objects can be partially hidden by other classes or random objects (sometimes only 10% are visible)
• if they are partially hidden, the visible part is contiguos, so no object is splitted apart
• the objects are stacked, so they will touch each other (maybe it's a stack of objects of the same class)

As the result I want to know how often the classes occurs on the image. Like class1: 0 times, class2: 3 times, class3: 1 time, ....

At first it sounds like a detection problem. But since the objects are stacked and yolov8 uses rectangles for training, 90% of the labels will contain (small) parts of other classes - or only very small fractions of the classes can be labeled. If using augmentation like rotation, it will be not possible to avoid class-mixtures in labeling.

What does it sounds like? Segmentation or detection. Are there pros/cons? I lean towards segmentation.

And: Does it make a big change for yolo if I use segmentation instead detection? I know, the pixels were classified but has the labeling process to be different? Thanks in advance

Answer 1

The scenario you're describing seems to initially suggest an object detection approach. However, given the complexity of objects being stacked and the necessity to count individual instances accurately, object detection alone, even with a robust model like YOLOv8, may not deliver the level of precision you need.

For greater accuracy, particularly where objects are partially obscured or in contact with one another, instance segmentation could be the more appropriate technique. Although it is more computationally intensive, instance segmentation models can provide the pixel-level detail required for precise identification and counting of object instances in challenging scenarios.

(source)

Answer 2

If I understand your requirement correctly, you're looking to tally the occurrences of each class in an image. I would suggest considering a detection approach as it might provide a more effective solution. Segmentation assigns labels to individual pixels, and in the presence of obstructions, it can be challenging to determine whether clusters of pixels belong to the same object or represent distinct instances of the same class.

Opting for a detection method offers additional advantages in your scenario. Since you lack a labeled dataset, the labeling process becomes more efficient with bounding box annotations compared to pixel-wise classification. Additionally, detection allows for accommodating obstructions by creating larger bounding boxes for obstructed objects.

I hope this perspective is helpful for your use case.