I have succesfully trained ssd_mobilenet_v2_keras for object detection, with a dataset of about 3700 images. Now I have more images to add. I tried adding only a few images (150-300) to see what happened, but what I obtain is that the trainig looks good in the first steps, but then there are some really high peaks in the loss function.
At first, I thought the problem was the quality of the pictures, so I removed them and tried to add more or less 300 bigger pictures: nothing changed. Then I tried to add only good pictures (no shadows or lights that may confuse the net, no interference with the object, only images where the objects I want to find are big and centered), but nothing.
All the things I have tried leads to the same results:

Loss peak in detail

End of training

As you can see, The training looks good at the beginning, but then there are those extremely high peaks that seems to happen at random steps (sometimes after 20.000 staps, sometimes after 2.000).
I tried to train both with and without some data augmentations (random contrast, brightness and saturtion adjust, random rgb-to-grayscale, random horizontal flip, ...) but the results are more or less the same (with data augmentations it's a little better, but still far from good).
Any suggestions on why this happens and how to fix?

EDIT: unfortunately I didn't take a screenshot at the end of the succesful training, I only have this one taken after 6.000 steps (total number of steps is 50.000), but then the chart followed this trend and ended with this values:
- classification loss: 4.16e-3
- localization loss: 1.11e-3
- total loss: 0.077

succesful training (partial results)

  • 1
    $\begingroup$ Could you share the loss graph before adding those images, so, how did the loss graph look before adding the images? I have a pretty good hypothesis of what is happening, but need more info $\endgroup$
    – JVGD
    Aug 20 at 9:35
  • $\begingroup$ @JVGD unfortunately I don't have the screenshot of the final result...I added one taken after 6.000 staps (total steps number is 50.000), I hope it is enough for you...let me know if you need more $\endgroup$
    – Deffo
    Aug 20 at 10:00
  • $\begingroup$ Ok, so the loss function is ok. I have seen the same in my experiments with custom loss functions that are not well defined or when data is not correctly annotated or not consistent. It might be happening that the new images you added to the dataset are not correctly annotated (maybe a parsing error or a format error) $\endgroup$
    – JVGD
    Aug 20 at 10:19
  • $\begingroup$ @JVGD mmm this sounds strange to me...I have personally annotated them, with the same method used for the 3400 starting images (I use the software labelImg, that generates .xml files). Also, I had added 300 pictures a couple of weeks ago (that's why in the question I said 3700) and I had no problems. From a quick check, the format looks the same...I don't think it's an annotation problem $\endgroup$
    – Deffo
    Aug 20 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.