2
$\begingroup$

I'm training a Tensorflow object detection model with approx. 7500 images of two classes, which contains approx. 10,000 classes per class. I'm using Tensorflow 2.6.0, in case that is relavent. I am using Single Shot Detector (with a ResNet 50 backbone). The image dimensions are 1024 x 1024, and the batch size is set to 2. Training is being done on Ubuntu 20.04 with a GeForce RTX 2080 Super (GPU).

After beginning training, the process is starting out at loss numbers to be expected:

INFO:tensorflow:{'Loss/classification_loss': 2.1305692,
 'Loss/localization_loss': 0.6402807,
 'Loss/regularization_loss': 1.407957,
 'Loss/total_loss': 4.178807,
 'learning_rate': 0.014666351}
I0903 16:56:21.947736 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 2.1305692,
 'Loss/localization_loss': 0.6402807,
 'Loss/regularization_loss': 1.407957,
 'Loss/total_loss': 4.178807,
 'learning_rate': 0.014666351}
INFO:tensorflow:Step 200 per-step time 0.447s
I0903 16:57:06.592366 140581900665792 model_lib_v2.py:698] Step 200 per-step time 0.447s
INFO:tensorflow:{'Loss/classification_loss': 1.2596315,
 'Loss/localization_loss': 0.6752764,
 'Loss/regularization_loss': 3.0123177,
 'Loss/total_loss': 4.9472256,
 'learning_rate': 0.0159997}
I0903 16:57:06.592768 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 1.2596315,
 'Loss/localization_loss': 0.6752764,
 'Loss/regularization_loss': 3.0123177,
 'Loss/total_loss': 4.9472256,
 'learning_rate': 0.0159997}
INFO:tensorflow:Step 300 per-step time 0.452s
I0903 16:57:51.830375 140581900665792 model_lib_v2.py:698] Step 300 per-step time 0.452s
INFO:tensorflow:{'Loss/classification_loss': 1.0455683,
 'Loss/localization_loss': 0.5895866,
 'Loss/regularization_loss': 3.0799737,
 'Loss/total_loss': 4.715129,
 'learning_rate': 0.01733305}
I0903 16:57:51.830749 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 1.0455683,
 'Loss/localization_loss': 0.5895866,
 'Loss/regularization_loss': 3.0799737,
 'Loss/total_loss': 4.715129,
 'learning_rate': 0.01733305}

Up until about step 16,800, the loss is decreasing to these numbers:

INFO:tensorflow:{'Loss/classification_loss': 0.5526215,
 'Loss/localization_loss': 0.28333753,
 'Loss/regularization_loss': 0.24686696,
 'Loss/total_loss': 1.0828259,
 'learning_rate': 0.037849143}
I0903 18:59:14.666097 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.5526215,
 'Loss/localization_loss': 0.28333753,
 'Loss/regularization_loss': 0.24686696,
 'Loss/total_loss': 1.0828259,
 'learning_rate': 0.037849143}
INFO:tensorflow:Step 16700 per-step time 0.446s
I0903 18:59:59.247199 140581900665792 model_lib_v2.py:698] Step 16700 per-step time 0.446s
INFO:tensorflow:{'Loss/classification_loss': 0.4649979,
 'Loss/localization_loss': 0.28323257,
 'Loss/regularization_loss': 0.2433301,
 'Loss/total_loss': 0.9915606,
 'learning_rate': 0.037820127}
I0903 18:59:59.247609 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.4649979,
 'Loss/localization_loss': 0.28323257,
 'Loss/regularization_loss': 0.2433301,
 'Loss/total_loss': 0.9915606,
 'learning_rate': 0.037820127}
INFO:tensorflow:Step 16800 per-step time 0.446s
I0903 19:00:43.835976 140581900665792 model_lib_v2.py:698] Step 16800 per-step time 0.446s
INFO:tensorflow:{'Loss/classification_loss': 0.43402833,
 'Loss/localization_loss': 0.1641234,
 'Loss/regularization_loss': 0.24129395,
 'Loss/total_loss': 0.8394457,
 'learning_rate': 0.03779093}
I0903 19:00:43.836373 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.43402833,
 'Loss/localization_loss': 0.1641234,
 'Loss/regularization_loss': 0.24129395,
 'Loss/total_loss': 0.8394457,
 'learning_rate': 0.03779093}

However, starting at about 16,900, the model total_loss rapidly increases, up to numbers even higher than are shown below:

INFO:tensorflow:Step 16900 per-step time 0.446s
I0903 19:01:28.390861 140581900665792 model_lib_v2.py:698] Step 16900 per-step time 0.446s
INFO:tensorflow:{'Loss/classification_loss': 0.5590624,
 'Loss/localization_loss': 0.5160909,
 'Loss/regularization_loss': 338.40286,
 'Loss/total_loss': 339.478,
 'learning_rate': 0.03776155}
I0903 19:01:28.391232 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.5590624,
 'Loss/localization_loss': 0.5160909,
 'Loss/regularization_loss': 338.40286,
 'Loss/total_loss': 339.478,
 'learning_rate': 0.03776155}
INFO:tensorflow:Step 17000 per-step time 0.445s
I0903 19:02:12.936022 140581900665792 model_lib_v2.py:698] Step 17000 per-step time 0.445s
INFO:tensorflow:{'Loss/classification_loss': 0.7908556,
 'Loss/localization_loss': 0.7274248,
 'Loss/regularization_loss': 858.3554,
 'Loss/total_loss': 859.87366,
 'learning_rate': 0.037731986}
I0903 19:02:12.936432 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.7908556,
 'Loss/localization_loss': 0.7274248,
 'Loss/regularization_loss': 858.3554,
 'Loss/total_loss': 859.87366,
 'learning_rate': 0.037731986}
INFO:tensorflow:Step 17100 per-step time 0.452s
I0903 19:02:58.127156 140581900665792 model_lib_v2.py:698] Step 17100 per-step time 0.452s
INFO:tensorflow:{'Loss/classification_loss': 0.7510178,
 'Loss/localization_loss': 0.49337074,
 'Loss/regularization_loss': 2617.2888,
 'Loss/total_loss': 2618.5332,
 'learning_rate': 0.03770224}
I0903 19:02:58.127575 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.7510178,
 'Loss/localization_loss': 0.49337074,
 'Loss/regularization_loss': 2617.2888,
 'Loss/total_loss': 2618.5332,
 'learning_rate': 0.03770224}
INFO:tensorflow:Step 17200 per-step time 0.445s
I0903 19:03:42.625258 140581900665792 model_lib_v2.py:698] Step 17200 per-step time 0.445s
INFO:tensorflow:{'Loss/classification_loss': 1.1258743,
 'Loss/localization_loss': 0.45634705,
 'Loss/regularization_loss': 394886900.0,
 'Loss/total_loss': 394886900.0,
 'learning_rate': 0.037672307}
I0903 19:03:42.625638 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 1.1258743,
 'Loss/localization_loss': 0.45634705,
 'Loss/regularization_loss': 394886900.0,
 'Loss/total_loss': 394886900.0,
 'learning_rate': 0.037672307}
INFO:tensorflow:Step 17300 per-step time 0.445s
I0903 19:04:27.112154 140581900665792 model_lib_v2.py:698] Step 17300 per-step time 0.445s
INFO:tensorflow:{'Loss/classification_loss': 0.57859087,
 'Loss/localization_loss': 0.53405523,
 'Loss/regularization_loss': 383440770.0,
 'Loss/total_loss': 383440770.0,
 'learning_rate': 0.037642203}
I0903 19:04:27.112533 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.57859087,
 'Loss/localization_loss': 0.53405523,
 'Loss/regularization_loss': 383440770.0,
 'Loss/total_loss': 383440770.0,
 'learning_rate': 0.037642203}

What could be the cause of this, and what would be the best way to go about fixing it?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

This is a classic "exploding gradient" problem. You can try adding gradient clipping or increasing the amount of regularization.

Decreasing the learning rate, or increasing the batch size may also help.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .