What is the basic difference between NAS, Hyperparameter Optimization/Tuning and Pruning?

Question

As far as I understand:

NAS = an algorithm that searches for the best NN architecture, i.e., how many layers, what activation function to use, how many neurons, etc.
Hyperparameter Optim = finding the best set of values for an already designed NN model.
Pruning = optimizing the connections between layers in a trained model.

My question is,

Luca Anzalone · Accepted Answer · 2023-04-15 07:37:11Z

Regarding question 1:

Use NAS when you don't know which is the best NN architecture for your problem. Basically, you start from a set of pre-defined blocks and NAS finds the best way to connect them and also their hyper-parameters like number of blocks, filters, etc.
As you said, hyper-opt is useful for an already designed NN (i.e. you know that such an architecture should work well) but also for the optimizer (e.g. learning rate, weight decay), and other hyper-parameters related to some custom layers, operation, loss. For example, in auto-encoders the size of the latent space is another hyper-parameter.
Finally, pruning is a model compression technique. So you use that to remove a given % of connections in your trained model: basically it sets some weights to zero. Pruning is done to get a lightweight version of your initial model, with the aim of deploying it on mobile, low-power devices and even FPGAs (field programmable gate arrays) boards. The pruned model has to be compiled by specific tools in order to get a practical advantage of the weights set to zero, to reduce the number of operations and also memory usage.

About question 2:

Well, I would use NAS first to determine the best NN architecture, then hyper-opt to find the best hyper-params of, e.g., the optimizer (and so not the ones related to the NN since NAS should have already found them), and finally, pruning as a post-training step but only if you want to deploy the model on some computing device with limited resources.

1 Answer 1