3
$\begingroup$

I'm working on semantic segmentation tasks in the medical space using the U-Net. Let's say that I train a U-Net model on medical images with the goal of segmenting out, say, ligaments, from a medical image. If I train that model on images that contain just a single labelled ligament, it will be able to segment out single ligaments pretty well, I assume. If I present it with an image with multiple ligaments, should it also be able to segment the multiple ligaments well too?

Based on my understanding, semantic segmentation is just pixel-wise classification. As a result, shouldn't the number of the objects in the image not be relevant since it's only looking at individual pixels? So, as long as a pixel matches that of a ligament, it should be able to segment it equally right?

Or am I misunderstanding some piece?

Basically, if I train a U-Net on images with just single ligaments, will it also be able to segment images with multiple ligaments equally as well based on my logic above?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
0
$\begingroup$

Without experimental evidence to back me up, I can not answer this with 100% confidence. However, I am fairly certain that this will cause issues depending on the model.

U-net is essentially an auto-encoder, and due to the fact that it is all just one big neural network, it is likely it will learn the easiest pattern (as all NN do), and that is to find one single instance of an object and shade that region.

Now why does this depend on the model? Well let's say you are using something slightly different, where region proposals are generated by a determinisitc algorithm we've predefined, then these regions are run through a CNN to segment them. In this case, as each region is without context of the entire image, the difference between 2 objects in an image and 1 is indistinguishable to the network (as regions may overlap), and as such, only using images with 1 object will not pose any problems (there is a name for early models like these, though it escapes me).

So assuming I am correct, what should you do? The models that use a deterministic algorithm for region proposals are slow and old, so I wouldn't suggest that. Instead, I would think that you should first do some testing, to see if it actually does cause issues. Assuming it does, a good option could be to tamper with the training data and separate segments by a few pixels to sort of "force" multiple objects into existence.

Regardless of such, I would still suggest using U-net. Fixing this issue (if it does arrise) should be relatively easy to do, so there's little to lose by using U-net and just trying the training.

Improve this answer
$\endgroup$
0
$\begingroup$

In the problem of Semantic Segmentation one solves the problem of investigating, whether a particular pixel on the image belongs to a certain class. For example, on the photo taken from the street one may be interested , whether this pixel belongs to the road, traffic light, sign, people or background of no interest.

Supeficially, in this setting it doesn't matter, whether there is a single instance of an object or multiple, and the thing, which is more important is the abundance of a particular class on an image.

However, CNN adopt to a particular patterns, pertinent to the distribution, from which the data comes from. It can be the case, that a network has captured some property and tooks a pattern, which is relevant only for one instance case. Images with multiple instances of class of interest are like outliers in the data, and may be treated due to this incorrecty.

I am not sure, whether it makes sense in yout particular problem, but maybe it is worth performing a mosaic augmentation https://towardsdatascience.com/data-augmentation-in-yolov4-c16bd22b2617. So that you will have for some samples of the augmented data a multiinstance case.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I think you misunderstood the question. The question is not about the difference between semantic and instance segmentation. The question is whether the model would generalize and be able to segment images with a different number of objects than the number of objects that can be found in the training data. I think the OP is not interested in knowing whether the model would be able to differentiate the instances of the same type of object (i.e. they were not asking about instance segmentation). So, I don't think that this answer really addresses the question. $\endgroup$
    – nbro
    Feb 7, 2021 at 12:46
  • $\begingroup$ @nbro I mean that superficially due to the absence of notion of single or multiple instances of some class the task is the same. However, if in the training data there are instances with only single intstance, and in the test data with multiple, it is likely to be treated as outlier, different from the distribution for which the NN was trained. Should I delete my answer? $\endgroup$
    – spiridon_the_sun_rotator
    Feb 7, 2021 at 13:51
  • $\begingroup$ Your answer contains useful information, but maybe you should also add some information that more directly addresses the question. So, I wouldn't delete your answer, if you plan to include this info. $\endgroup$
    – nbro
    Feb 7, 2021 at 13:53
0
$\begingroup$

I'm not aware of a model that could generalize from single segment to multi segment without nontrivial surgery, augmentation, and retraining, but I have had some success doing this in cases where the regions to segment are non-overlapping: After finding the first segment, edit the input image to distort or obscure the segmented region and run it through again, repeating until it stops finding new segments.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .