1
$\begingroup$

Multi-class classification is simply assigning all data points into one of up to any finite number of mutually exclusive labels. I am new to the field(s) of AI/ML and I keep hearing people use the term "semantic segmentation."

I want to "translate" this AI/ML jargon into something more familiar to me. The best video I have found so far to explain what it is made me wonder, what is the difference between semantic segmentation and classification?

NOTE: I am specifically not referring to so-called multi-label "classification" which allows a data point to have more than one label at a time. In my experience, that sort of labeling is not classification at all, which is a division into mutually exclusive sets (no overlap).

Improve this question
$\endgroup$
0

3 Answers 3

Reset to default
1
$\begingroup$

Image/object classification (or recognition)

(Multi-class) image/object classification (or recognition) typically refers to the task of assigning one label to an image, so we typically assume that there's only one main object in the image. The multi-class only refers to the fact that we have more than 2 possible classes or labels (if we had only 2, this would be binary classification), but note that this does not mean that we have more than one main object in each image. So, in this task, we are not interested in labelling each pixel, but to label the whole image, so, in this sense, this is a sparse classification task. An example of a dataset that is used for image classification is MNIST, where there's only one object (a number) per image. Here's a picture that shows 3 MNIST images, each of them has only one associated label (below), which corresponds to the number in the image.

enter image description here

Semantic segmentation

Semantic segmentation is the task of classifying each pixel in an image (or at least groups of pixels), so that objects of different classes have their pixels labelled differently. Instance segmentation is a similar task, but we additionally want to differentiate between different objects of the same class, so we assume that there could be more than one object in the image and there could even be more objects of the same type/label. Given that we label pixels, this is a dense classification task.

Here's an example of an image that has been segmented, i.e. pixels associated with the same object (e.g. the umbrella) have the same label (color).

enter image description here

Object detection

So, in this way, semantic (or instance) segmentation is more similar to object detection, which is both a classification and regression task, because we want both to classify one or more objects in the image, but we also want to draw a bounding box around them (and this is often solved as a regression problem). The reason why we draw a bounding box around each object is that, as opposed to image/object classification, there can be more than one object in the image, so we need a way to identify the locations of the objects. As opposed to semantic/instance segmentation, this is also a sparse classification task.

Here's an image to which object detection has been applied.

enter image description here

Improve this answer
$\endgroup$
5
  • $\begingroup$ I was hoping for a straightforward answer, which is lacking. $\endgroup$
    – tdMJN6B2JtUe
    Commented Jun 22, 2021 at 14:18
  • 1
    $\begingroup$ @brethvoice If you ignore the "object detection" section (which was not part of your question, but I decided to add it because I think it can be useful) and the images, my answer is about 20 lines long. I don't know how to convey the definitions of these tasks in a shorter but still understandable way. $\endgroup$
    – nbro
    Commented Jun 23, 2021 at 10:18
  • 1
    $\begingroup$ @brethvoice Maybe this description is what you are looking for: Object classification = sparse classification + typically only one main object in the image + each image is labelled with the only main object in the image. Image segmentation: dense classification (you classify pixels) + multiple objects in the image. In any case, the images above were supposed to intuitively convey the meaning of these tasks. $\endgroup$
    – nbro
    Commented Jun 23, 2021 at 10:20
  • $\begingroup$ Here is what I am getting at: "objects of different classes have their pixels labelled differently" does not preclude a pixel from having more than one label simultaneously, does it? $\endgroup$
    – tdMJN6B2JtUe
    Commented Jun 23, 2021 at 11:59
  • 1
    $\begingroup$ @brethvoice As far as I know, yes, it generally precludes a pixel from having more than one label, because, otherwise, it would be a multi-label classification problem. Of course, it may also be possible to formulate image segmentation so that you associate multiple labels to the same pixel, if that makes sense for your problem (not sure when that would be needed, though). $\endgroup$
    – nbro
    Commented Jun 23, 2021 at 21:28
0
$\begingroup$

Both things are similar. But, I think there is a bit of a difference in interpretation.
If what you are solving is a multi-class classification problem in an image, a proper measure of performance of an algorithm would be the accuracy of the prediction for each pixel.
While one of the most used measures of performance for semantic segmentation is the IOU (intersection over union) for each class. Which, makes sense if your objective is to create a segmentation (a mask) for each class.

Improve this answer
$\endgroup$
0
$\begingroup$

Why are you asking about 3 unrelated items? Multi-class and mutually exclusive are related. Semantic segmentation is an approach to object recognition in images, that has nothing to do with the other 2.

The different between multi-class and mutually exclusive is simple: a spam engine is an example of mutually exclusive, each item either is spam or is not. An algorithm that would potentially guess what grade an essay would get is an example of multi-class: there are many potential classifications, not a simple binarism.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .