When training relatively small DL model which takes several hours to train, I typically start with some starting points from literature and then use trial-and-error or grid-search approach to tune up hyperparameter values in order to preventing overfitting and achieve sufficient performance.

However, it is not uncommon for large models to have training time measured in days or weeks [1], [2], [3].

How are hyperparameters determined in such cases?

  • $\begingroup$ I don't get the issue: When you tell models have to be trained for days and weeks, you know how the hyperparameters are obtained? Through training..? $\endgroup$ – Ben Jul 24 at 14:05
  • 1
    $\begingroup$ I've seen some papers which are deriving a small dataset which has more or less the same characteristics as the large one and do the hyper-parameter selection on it. $\endgroup$ – razvanc92 Jul 24 at 14:10

In general, it is definetely very computationally expensive, so the exhaustive search is not perfromed in pratice, however, there are some recent approaches for determining, whether the architecture is fine, without performing the training - by looking at the covariance matrix after forwarding the data, for example, in a recent paper - https://arxiv.org/abs/2006.04647. However such approach is limited by far.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.