5
$\begingroup$

How do you efficiently choose the hyper-parameters of a neural network (e.g. the learning rate, number of layer, weights, etc.)?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Are you asking which techniques exist to search for the best combination of hyper-parameters? $\endgroup$
    – nbro
    Commented Feb 27, 2019 at 16:43

1 Answer 1

Reset to default
1
$\begingroup$

In deep learning era, there are two possible choices. Caveat approach and Panda approach.

Caviar Approach

In this approach, it is supposed that you have a very powerful cluster system that enables you to run different models simultaneously on different nodes. In this manner, you construct a d-dimensional space which each corresponds to a special hyperparameter. Then you can have a grid approach to partition the space and for each intersection of the grids, you can have a possible initialisation although there are some better ways in order to find better initialisations.

Panda Approach

This approach is widely used among students due to using desktop computers. In this approach, you usually try to find an initialisation of hyperparameters based on your experience or the other architectures' which are available and you try to refine them step by step by using cross-validation.

To answer your question, the efficiency depends on your computing power. For more details, you can take a look at the contents of the third week.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I have never heard of these terms/expressions. Where did you hear them? In general, you don't need a cluster. It depends on your NN model. A GPU (or even a CPU) will suffice if you have a simple NN model. It also depends on your time requirements. Furthermore, note that there are other hyper-parameter techniques other than grid search. $\endgroup$
    – nbro
    Commented Feb 27, 2019 at 16:37
  • $\begingroup$ I added the link. You can't train a YOLO model on a single GPU. $\endgroup$
    – Green Falcon
    Commented Feb 27, 2019 at 16:44
  • $\begingroup$ The question is not clear. Maybe the OP is asking for techniques to look for combinations of hyper-parameters. In that case, your answer is not valid. I would first ask for clarifications. $\endgroup$
    – nbro
    Commented Feb 27, 2019 at 16:46
  • $\begingroup$ Combination of hyperparameters? Isn't it exactly what I've tried to convey? $\endgroup$
    – Green Falcon
    Commented Feb 27, 2019 at 16:49
  • $\begingroup$ No, I'm talking about something different... $\endgroup$
    – nbro
    Commented Feb 27, 2019 at 16:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .