I have images that contain lots of elements. Some I know, some I don't. I want to know if it's ok to only label those I do know. Let's take this image for example. I would label the green stuff and the worm but leave the rest unlabeled. Is that ok?

Another question I would also like to ask is how concise I should be in labeling. For instance, You can see in the picture a bit of blue behind the green plant. So should I label that bit and say water or leave it unlabeled?


I also want to ask if it's ok to label only the things I'm interested in even if they take up to 30% of the picture? Won't the neural network be confused by all the details in the picture that it perceives and that I label as A for example, even if A is just a part of it?

Another question would be, let's say I have labels A, B and C. I have an image in which I'm a bit confused if a certain object is of label B or A or even a totally different class other than (A,B,C). What should I do in this instance?

I'm having a hard time with the dataset. It would take an expert to label this correctly. But I want to do things as cleanly as possible, so all the effort doesn't go to waste. I would really appreciate your help. Thank you guys.

enter image description here

  • $\begingroup$ Are you doing classification or object detection? $\endgroup$ – Sharan Jan 11 '20 at 11:21
  • $\begingroup$ @Sharan Hi I'm doing classification $\endgroup$ – user Jan 11 '20 at 11:57

You can't label things you don't know. The goal of labeling is to label the things you want the classifier to learn so that when you run it in inference mode you can discover what is in your data (new data that you didn't use for training, validating, or testing).

It is not a good idea to label small objects like the 'blue water' unless it is important to you to discover these fine details in inference mode.

  • $\begingroup$ Thank you for your answer. Can I ask if you previously worked on multi-label classification and what metrics you used if you did? $\endgroup$ – user Jan 12 '20 at 11:00

I think what you are actually talking about is semantic segmentation (where you label pixels individually).

There is a difference in theses tasks like Classification, Detection or Semantic Segmentation.

Classification refers to the task of giving a (usually) single label to the whole image, e.g. cat. But as you already noticed this does not nececerraly end in a clear labeling policy since you basically always have multiple classes in one image. However a ANN usually learns the most relevant (biggest, nearest) object in an image to set it to the corresponding class (but this of course again depends on how the images are labeled). At inferencing of the ANN you then get a probability distribution over all predefined classes. You can of course use this to take the K most relevant classes instead of just the single most relevant classes, to cover cases where multiple objects a probably present. However common output layers, e.g. like Softmax, are usually designed to favour for one single class instead of multiple classes, so you should keep that in mind or consider using a better suited output layer function for your use-case.

To have a more general apporach in the task of object detection you classify multiple objects in an image (usually as bounding boxes). That means you label all predefined objects with their position and class in an image.

And the most general approach here would be to do semantic segmentation that labels every pixel to a corresponding class, which then gives you the actual object borders etc. You can also label pixels as "voids" (or something like that) to cover unknown classes or classes that are not considered in your dataset. However creating such dataset is a horrible amount of work.

To clear this up. Look at your actual use-case and think about what you actually want your neural network to do. According to this you then think of a labeling policy and label your data.

  • $\begingroup$ Thank you. I originally wanted to do semantic segmentation. However, I don't know many of the objects in the picture and I also would need a lot of time to prepare the data. So I felt like multi-label was a decent approach to consider. $\endgroup$ – user Jan 12 '20 at 11:05
  • $\begingroup$ Okay. So for multi-label I think its best to use a normal sigmoid function on your final output layer and then threshold on the class outputs. The typical used softmax layer at output layer in general classification tasks tries to favour one class due to the normalization over all possible classes. So this might lead to unwanted behaviours. $\endgroup$ – Marcel_marcel1991 Jan 15 '20 at 21:13

I would classify each pixel separately instead of giving a label to the whole image. Sadly preparing the training data is very tedious and time-consuming.

Let's say the input image has dimensions of 200 x 300 x 3 (RGB) and there are two classes of regions you want to identify. A few approaches come to mind:

1) Train two separate networks, each forecasting a binary mask of size 200 x 300 of the object class in guestion.

2) Train a single network with a binary output of size 200 x 300 x 2 (sigmoid activation)

3) Train a single network with a binary output of size 200 x 300 x 3 (softmax activation), the 3rd class is for "other"

If you are uncertain of some regions you can leave its class probability to 50% and it won't affect cross-entropy losses.

Option 1 is easiest to get started with but training a single network should be more computationally more efficient than training two separate ones. In addition options 1 and 2 can forecast a single pixel belonging to both classes with 100% probability unlike the network of option 3.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.