How do you efficiently choose the hyper-parameters of a neural network (e.g. the learning rate, number of layer, weights, etc.)?

  • 1
    $\begingroup$ Are you asking which techniques exist to search for the best combination of hyper-parameters? $\endgroup$ – nbro Feb 27 '19 at 16:43

In deep learning era, there are two possible choices. Caveat approach and Panda approach.

Caviar Approach

In this approach, it is supposed that you have a very powerful cluster system that enables you to run different models simultaneously on different nodes. In this manner, you construct a d-dimensional space which each corresponds to a special hyperparameter. Then you can have a grid approach to partition the space and for each intersection of the grids, you can have a possible initialisation although there are some better ways in order to find better initialisations.

Panda Approach

This approach is widely used among students due to using desktop computers. In this approach, you usually try to find an initialisation of hyperparameters based on your experience or the other architectures' which are available and you try to refine them step by step by using cross-validation.

To answer your question, the efficiency depends on your computing power. For more details, you can take a look at the contents of the third week.

  • $\begingroup$ I have never heard of these terms/expressions. Where did you hear them? In general, you don't need a cluster. It depends on your NN model. A GPU (or even a CPU) will suffice if you have a simple NN model. It also depends on your time requirements. Furthermore, note that there are other hyper-parameter techniques other than grid search. $\endgroup$ – nbro Feb 27 '19 at 16:37
  • $\begingroup$ I added the link. You can't train a YOLO model on a single GPU. $\endgroup$ – Media Feb 27 '19 at 16:44
  • $\begingroup$ The question is not clear. Maybe the OP is asking for techniques to look for combinations of hyper-parameters. In that case, your answer is not valid. I would first ask for clarifications. $\endgroup$ – nbro Feb 27 '19 at 16:46
  • $\begingroup$ Combination of hyperparameters? Isn't it exactly what I've tried to convey? $\endgroup$ – Media Feb 27 '19 at 16:49
  • $\begingroup$ No, I'm talking about something different... $\endgroup$ – nbro Feb 27 '19 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.