0
$\begingroup$

I'm kind of stuck, and instead of trying to randomly shoot the net with my ideas maybe I can consult it with you (one epoch takes 7h, so I cant't test my random ideas). Here's the crime scene:

  1. My objective is to train a VGG-family net on specific custom moderately-large dataset (4.3 mln images, 7205 classes).
  2. Since 1 epoch takes 7h to calculate (on whole dataset), I've tuned the hyperparameters on 300 classes (approx. 200 000 images). The net gets about 50% top-1 accuracy after 40 epochs, which is ok for me (learning curve attached), and also does pretty good on 1000 classes (on the pic).
  3. Now I'm prepared for the big heist and I'm training the net on whole dataset. But the net doesn't really learn anything, with the accuracy oscillating around random, even after 44 epochs (yep, 13 days of training).
  4. Specs:
    • batch size: 64
    • learning rate: log decay from 0.01
    • class weightening to prevent overrepresented classes to mess with weights
    • net trained from scratchenter image description here

The net architecture and training is exactly the same (except ofc last FC layer which size I've changed from 300 to 7205). Do you have any ideas on this? To small FC layers? Wrong learning rate? What Am I missing?

|improve this question
$\endgroup$
  • $\begingroup$ It would be insightful to look at the loss curve. Did you try dropout on FC layers? Not much sure if this initial learning rate is high (may be), I have seen a few deep CNNs, one being this VGG16 I trained, for 1e-3 $\endgroup$ – Karan Shah Jun 8 at 6:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.