I'm kind of stuck, and instead of trying to randomly shoot the net with my ideas maybe I can consult it with you (one epoch takes 7h, so I cant't test my random ideas). Here's the crime scene:
- My objective is to train a VGG-family net on specific custom moderately-large dataset (4.3 mln images, 7205 classes).
- Since 1 epoch takes 7h to calculate (on whole dataset), I've tuned the hyperparameters on 300 classes (approx. 200 000 images). The net gets about 50% top-1 accuracy after 40 epochs, which is ok for me (learning curve attached), and also does pretty good on 1000 classes (on the pic).
- Now I'm prepared for the big heist and I'm training the net on whole dataset. But the net doesn't really learn anything, with the accuracy oscillating around random, even after 44 epochs (yep, 13 days of training).
The net architecture and training is exactly the same (except ofc last FC layer which size I've changed from 300 to 7205). Do you have any ideas on this? To small FC layers? Wrong learning rate? What Am I missing?