I have read a lot about NAS, but I still do not understand one concept: When setting up a neural network, hyperparameters (such as the learning rate, dropout rate, batch size, filter size, etc.) need to be set up.

In NAS, only the best architecture is decided, e.g. how many layers and neurons. But what about the hyperparameters? Are they randomly chosen?


It's not clearly stated (it's not stated at all on Wikipedia), but, after a bit of searching, I found an answer here about a third of the way down the page:

The best performing architecture observed during the training of the controller is taken, and a grid search is performed over some basic hyperparameters such as learning rate and weight decay in order to achieve near STOTA (state of the art) performance.

So, as a direct answer: The norm; A grid search.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.