When training relatively small DL model which takes several hours to train, I typically start with some starting points from literature and then use trial-and-error or grid-search approach to tune up hyperparameter values in order to preventing overfitting and achieve sufficient performance.

However, it is not uncommon for large models to have training time measured in days or weeks [1], [2], [3].

How are hyperparameters determined in such cases?

When training a relatively small DL model, which takes several hours to train, I typically start with some starting points from literature and then use a trial-and-error or grid-search approach to fine-tune the values of the hyper-parameters, in order to prevent overfitting and achieve sufficient performance.

However, it is not uncommon for large models to have training time measured in days or weeks [1], [2], [3].

How are hyperparameters determined in such cases?