Getting worse performance when training a pre-trained model with the existing class

Question

I am training pre-trained SSD-InceptionV2-Coco to detect the "car",

which is one of the classes in mscoco label.

I train the model with ~50k sample from KITTI, 500k iteration with batch size 2.

I followed this script to generate tfrecord file.

Then I test both original pre-trained model and my trained model with one video.

The performance of my trained model is worse. More missing detected results.

One thing I found recently is the classification_loss/localization_loss increases when AvgNumGroundtruthBoxesPerImage increases.

EDIT

Another thing I found is the more ground truth boxes per image I have,

the less average num positive anchors per image I have.

This bothers me because if the number of anchors generated per image is fixed,

more ground truth boxes should provide more positive anchors per image.

So I wonder where to find the root cause.

Any suggestion is welcome.

Thank you for precious time on my question.