I have succesfully trained ssd_mobilenet_v2_keras for object detection, with a dataset of about 3700 images. Now I have more images to add. I tried adding only a few images (150-300) to see what happened, but what I obtain is that the trainig looks good in the first steps, but then there are some really high peaks in the loss function.
At first, I thought the problem was the quality of the pictures, so I removed them and tried to add more or less 300 bigger pictures: nothing changed. Then I tried to add only good pictures (no shadows or lights that may confuse the net, no interference with the object, only images where the objects I want to find are big and centered), but nothing.
All the things I have tried leads to the same results:
As you can see, The training looks good at the beginning, but then there are those extremely high peaks that seems to happen at random steps (sometimes after 20.000 staps, sometimes after 2.000).
I tried to train both with and without some data augmentations (random contrast, brightness and saturtion adjust, random rgb-to-grayscale, random horizontal flip, ...) but the results are more or less the same (with data augmentations it's a little better, but still far from good).
Any suggestions on why this happens and how to fix?
EDIT: unfortunately I didn't take a screenshot at the end of the succesful training, I only have this one taken after 6.000 steps (total number of steps is 50.000), but then the chart followed this trend and ended with this values:
- classification loss: 4.16e-3
- localization loss: 1.11e-3
- total loss: 0.077