I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read that doing a grid search for hyperparameters is not the best way to go about training, and that random search is better in this case. Is random search really that good?

New contributor
user2651062 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • $\begingroup$ 1. Learning rate schedule (decrease rate, cyclic etc) 2.Momentum if used(including 0) 3. augmentation (random transformations values) if used $\endgroup$ – mirror2image Jan 15 at 15:03
  • $\begingroup$ @mirror2image momentum is fixed to 0.9 (it appears that it's common practice to fix it to this value if you want to focus on other hyperparameters). Augmentations are done, but I'm mostly talking about other hyperparameters : regularizarion, weight decay... AND MOST IMPORTANTLY: from where to start $\endgroup$ – user2651062 Jan 15 at 15:14
  • $\begingroup$ Weight decay is useless in most cases. There are a lot of different regularizations, use common sense for them. Fixed momentum is bad practice. Always good idea to check momentum 0, 0.6, 0.9 Start with different learning rate schedules $\endgroup$ – mirror2image Jan 15 at 15:18
  • $\begingroup$ @mirror2image so you say: I start with trying out different lr scheduling policies, and during each policy, I do a grid search of the learning rate, momentum and the parameters of regulatization ? $\endgroup$ – user2651062 Jan 15 at 16:32
  • $\begingroup$ First - schedule and learning rate with grid search. After that momentum and regularization separately - it's highly likely they are independent. $\endgroup$ – mirror2image Jan 15 at 17:55

Your Answer

user2651062 is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.