For a client I needed to figure out if a machine learning model can classify plastic container loading patterns. These containers are delivered on pallets in varying patterns.
So I installed a camera setup that takes images of incoming pallets and saves these images together with the correct label. The region of interest is always in the exact same place in the image and ~40% of the image is background.
After training a model with ML.NET's modelbuilder on a dataset of 7000 images divided over six classes I get a model that's 97% accurate, not bad.
I made a test setup using the code generated by ML.NET in the garage with a pallet and said containers. When testing the model in this setup, the results are disappointing, always predicting the same (wrong) class and with scores as high as 0.999. The test setup is almost identical to the setup that made the training data, except for the background and maybe the ambient lichting.
Is the background the problem here? Can I solve this with data augmentation? (vertical mirroring, trans, rotate and scale a bit) Should I write a script to crop the images so mostly the ROI is left? Or is the data I generated useless?
Now you're thinking why didn't he include some images? Well, my client doen't allow it. Blacking out sensitive areas would leave the image 95% black. But here is a paint doodle of the situation if that helps.
So to clarify the only thing that varies in the 7000 images is the content on the pallet, it varies in pattern and height. The background of the training data never varies and has lots of features.
EDIT: What I can show is some of the cropped and high-pass filtered data that I used to train the model, one closed example and one open.