How to correctly label images for multi-label classification?

Question

I have images that contain lots of elements. Some I know, some I don't. I want to know if it's ok to only label those I do know. Let's take this image for example. I would label the green stuff and the worm but leave the rest unlabeled. Is that ok?

Another question I would also like to ask is how concise I should be in labeling. For instance, You can see in the picture a bit of blue behind the green plant. So should I label that bit and say water or leave it unlabeled?

EDIT:

I also want to ask if it's ok to label only the things I'm interested in even if they take up to 30% of the picture? Won't the neural network be confused by all the details in the picture that it perceives and that I label as A for example, even if A is just a part of it?

Another question would be, let's say I have labels A, B and C. I have an image in which I'm a bit confused if a certain object is of label B or A or even a totally different class other than (A,B,C). What should I do in this instance?

I'm having a hard time with the dataset. It would take an expert to label this correctly. But I want to do things as cleanly as possible, so all the effort doesn't go to waste. I would really appreciate your help. Thank you guys.

Answer 1

You can't label things you don't know. The goal of labeling is to label the things you want the classifier to learn so that when you run it in inference mode you can discover what is in your data (new data that you didn't use for training, validating, or testing).

It is not a good idea to label small objects like the 'blue water' unless it is important to you to discover these fine details in inference mode.

Answer 2

I think what you are actually talking about is semantic segmentation (where you label pixels individually).

There is a difference in theses tasks like Classification, Detection or Semantic Segmentation.

Classification refers to the task of giving a (usually) single label to the whole image, e.g. cat. But as you already noticed this does not necessarily end in a clear labeling policy, since you basically always have multiple classes in one image. However, a ANN usually learns the most relevant (biggest, nearest) object in an image to set it to the corresponding class (but this of course again depends on how the images are labeled). At inferencing of the ANN, you then get a probability distribution over all predefined classes. You can, of course, use this to take the K most relevant classes instead of just the single most relevant classes, to cover cases where multiple objects are probably present. However common output layers, e.g. like Softmax, are usually designed to favour for one single class instead of multiple classes, so you should keep that in mind or consider using a better suited output layer function for your use-case.

To have a more general approach in the task of object detection you classify multiple objects in an image (usually as bounding boxes). That means you label all predefined objects with their position and class in an image.

And the most general approach here would be to do semantic segmentation that labels every pixel to a corresponding class, which then gives you the actual object borders etc. You can also label pixels as "voids" (or something like that) to cover unknown classes or classes that are not considered in your dataset. However creating such a dataset is a horrible amount of work.

To clear this up: look at your actual use-case, and think about what you actually want your neural network to do. According to this, you then think of a labeling policy and label your data.

Answer 3

I would classify each pixel separately instead of giving a label to the whole image. Sadly preparing the training data is very tedious and time-consuming.

Let's say the input image has dimensions of 200 x 300 x 3 (RGB) and there are two classes of regions you want to identify. A few approaches come to mind:

1) Train two separate networks, each forecasting a binary mask of size 200 x 300 of the object class in guestion.

2) Train a single network with a binary output of size 200 x 300 x 2 (sigmoid activation)

3) Train a single network with a binary output of size 200 x 300 x 3 (softmax activation), the 3rd class is for "other"

If you are uncertain of some regions you can leave its class probability to 50% and it won't affect cross-entropy losses.

Option 1 is easiest to get started with but training a single network should be more computationally more efficient than training two separate ones. In addition options 1 and 2 can forecast a single pixel belonging to both classes with 100% probability unlike the network of option 3.