0
$\begingroup$

For a client I needed to figure out if a machine learning model can classify plastic container loading patterns. These containers are delivered on pallets in varying patterns.

So I installed a camera setup that takes images of incoming pallets and saves these images together with the correct label. The region of interest is always in the exact same place in the image and ~40% of the image is background.

After training a model with ML.NET's modelbuilder on a dataset of 7000 images divided over six classes I get a model that's 97% accurate, not bad.

I made a test setup using the code generated by ML.NET in the garage with a pallet and said containers. When testing the model in this setup, the results are disappointing, always predicting the same (wrong) class and with scores as high as 0.999. The test setup is almost identical to the setup that made the training data, except for the background and maybe the ambient lichting.

Is the background the problem here? Can I solve this with data augmentation? (vertical mirroring, trans, rotate and scale a bit) Should I write a script to crop the images so mostly the ROI is left? Or is the data I generated useless?

Now you're thinking why didn't he include some images? Well, my client doen't allow it. Blacking out sensitive areas would leave the image 95% black. But here is a paint doodle of the situation if that helps.

So to clarify the only thing that varies in the 7000 images is the content on the pallet, it varies in pattern and height. The background of the training data never varies and has lots of features.

MsPaint example of situation

EDIT: What I can show is some of the cropped and high-pass filtered data that I used to train the model, one closed example and one open.

Closed containers Open containers

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If the background and the ambient light are the problem, I suggest you to agument your training data by introducing that kind of variance there. Just mirroring and rotating might not be sufficient, since they have little effect on the background's texture etc.

Alternatively you'll need to collect a more realistic dataset for training.

Cropping is a great idea, that way there is less chance for the network to be mislead by irrelevant changes in the input.

An other idea is to try filtering the image before passing it to the network, for example with a high-pass filter. This way the large-scale gradients caused by ambient light sources don't show up as strongly. The AI should focus on sharp edges of the boxes anyway, not their color. Your example image shows this point very well :)

And more thing to try is to make your network much simpler, to avoid overfitting.

Now that you have more images, you can add them to your validation set and immediately see whether you are making progress.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Far fetched idea: Would it be possible to train a model to semantically segment the ROI's (which are easily distinguishable), and with that removing the background? This could be applied to the full dataset to generate a dataset only containing ROI.. To prevent the black background from becoming a feature I could generate random backgrounds from images from other unrelated datasets available on the internet. $\endgroup$
    – user44299
    Nov 17, 2022 at 10:02
  • $\begingroup$ That is a good pre-processing step, and great if you can automate that by training a model to do it. But I don't quite understand why would you add random backgrounds, after having just removed it? Is this for data augmentation, and the final network would run without the ROI identification? Here assume the whole palled would be the ROI, not individual boxes. $\endgroup$
    – NikoNyrh
    Nov 17, 2022 at 23:33
  • $\begingroup$ And looking at your illustration from a fresh perspective, it might be worthwhile to try identifying the borders which separate different boxes from each other. If you can get this to work, it isn't too much effort to determine how many connected components (boxes) there are. The output is also easier to debug, rather than a plain classification. Here it is worth noting that down and right facing walls must be ignored, we are interested only of box faces facing up! Hey maybe that could be the ROI. And this should be a totally separate answer, to allow up & downvoting. $\endgroup$
    – NikoNyrh
    Nov 17, 2022 at 23:36
  • $\begingroup$ I thought adding a non static background would make it generalize better.. Detecting the individual boxes is an interesting idea, maybe using instance segmentation, unfortunately the containers can differ very much and can even be open on top containing random parts. I trained a model with my previous plans (no background, color/high pass filters, skew, scale, rotate, flip), it performs horribly on test set as well as original images, I'm so very stuck now. $\endgroup$
    – user44299
    Nov 22, 2022 at 9:16
  • $\begingroup$ Since you cannot share any of the original images, I cannot really try out any of my ideas either. But actually if your network is performing poorly on for example high-pass filtered images, I see it as a sign of progress! If a human can still solve the problem with augmented images, so should a neural network. And if you are lucky, once you get it to work on the training set it will also work on the test set. Remember to apply the filter on all images, not just some of them, to be consistent. $\endgroup$
    – NikoNyrh
    Nov 22, 2022 at 19:33

You must log in to answer this question.