I fine tuned MobileNetSSD for object detection using a dataset with just one class (~4000 images). All the training images include at least one bounding box related to that class (no empty images). By following the example with the VOC dataset, the labelmap includes two classes, the background and my custom class. However, as I mentioned, there are no annotations related to the background and I am not sure if there should be any.
Now my fine tuned network performs very well when objects belonging to my class are present, however there are some false detections with very high confidence when the class is not present. Can this be related to the fact that I don't have empty images in my training set?