# What make a CNN suitable for image classification or for semantic segmentation?

I've just started with CNN and there is something that I haven't understood yet:

How do you "ask" a network: "classify me these images" or "do semantic segmentation"?

I think it must be something on the architecture, or in the loss function, or whatever that makes the network classify its input or do semantic segmentation.

I suppose its output will be different on classification and on semantic segmentation.

Maybe the question could be rewritten to:

What do I have to do to use a CNN for classification or for semantic segmentation?

Disclaimer: This question is very broad, my answer is admittedly partial and is intended to just give an idea of what's out there and how to find out more.

How do you "say" a network: "classify me these images" or "do semantic segmentation"?

You're mixing two very different problems there. Although there are SO many variations of problems people are applying CNNs to, for this example we can focus on the "classification of something in the image" subset ad identify 4 key tasks:

• Image Classification answers the question "What is this image about" (the answer is an image where each pixel is assigned to one of the given classes)
• Semantic Segmentation answer he question "What areas of this image are part of a Cat?" (e.g.,
• Object Detection answers the question "Where are the objects in the image AND what objects are they?" (e.g. of answer: "Cat in bounding box at x,y,w,h [10,20,50,60]")
• Semantic Segmentation answers the question "Where are the individual objects in this image AND what class are they AND give me the pixels that belong to each object". You may guess from the number of ANDs there, this is the hardest of the four. The output here would be a set of class, bounding_box, mask tuples where the mask is typically defined in relation to the returned bounding box.

So, how do we build networks capable of solving one problem or the other? We build architectures towards one specific problem, exploiting reusable parts where possible. For example, typically classification and object detection are based on a deep "backbone" that extracts highly complex features from the image, that finally are used by a classifier layer interprets to make a prediction (for image classification) or a box prediction head to predict where objects lie in the image (very big simplification, look up object detection architectures and how they work for the proper description!).

What do I have to do to use a CNN for classification or for semantic segmentation?

In principle you can't just take a network built for classification and just "ask" it to do semantic segmentation (think of it as trying to use a screwdriver as scissors... it just was not built for that!). You need changes in the architecture, which necessarily imply new training, at the very least for the new parts that were added.