I am using Tensorflow Object Detection API for training a CNN from scratch on COCO dataset. I need to use this specific configuration. There is no pre-trained model on COCO with that configuration and this is the reason why I am training from scratch.
However, after 1 week of training and evaluating each checkpoint generated by the training phase this is how my learning phase appears on Tensorboard:
Thus, my questions are:
- does anyone know how many iterations approximately will be necessary? Right now I did more than 500'000 iterations.
- How can be possible that after 500'000 the evaluation is 0,8%? I would expected something like 60-70%.
- Why does there is a sudden drop after 500k iterations? I thought that the eval was supposed to converge to some limit. (this is what SGD should do)
- Is there any 'trick' to speed up the training phase? (ex: increasing the learning rate, etc).