Hyperparameter tuning methods for neural networks

Question

I have a fully connected feedforward classifier neural network that uses the leaky ReLU activation function. I would like to apply a state-of-the-art hyperparameter tuning method to my methodology. Currently, the fitness function associated with each parameter setting is a weighted combination of (i) accuracy of both training and test data sets ($acc_{tr}$ and $acc_{ts}$, respectively), (ii) numbers of activated hidden layers ($nL$) and their neurons ($nN_l$), and (iii) number of epochs ($nE$) - length of training. The fitness function is computed as follows:

\begin{equation}\label{eqq} FF = 0.7 \times (acc_{tr} + acc_{ts}) + 0.1 \times \frac{nL}{10} + 0.1 \times \frac{\sum_{l \in \mathcal{L}} nN_l}{1000} + 0.1 \times \frac{nE}{2000} \end{equation}

Note that I have determined the weights in function $FF$ based on my experimental observations. Potential values for the parameters of this neural network are given in the following table:

$\alpha$ here is the leakage parameter of the ReLU activation function. Considering that I am new to this field, my questions are as follows:

• Is the weighted fitness function an appropriate one? Should I include both training and test accuracies at the same time? Why?
• Do the potential values make sense for a fully connected feedforward classifier neural network with the leaky ReLU activation function? Are there any other parameters that I have to tune? Please consider that I have developed this neural network using the Keras library.
• The above table also shows the elite configuration. I have found this parameter setting using the iRace package. Do you recommend any other method for hyperparameter tuning in neural networks?
• I have seen many papers use genetic algorithms to tune their parameters. What are the advantages of using a genetic algorithm in comparison to the iRace package (if there is any)?

Please kindly answer my above questions with supporting scientific references. Also, please let me know if there are any other considerations for the hyperparameter tuning of neural networks. Thank you!

Answer 1

There is a Python package called Tune is fairly easy to use, and contains most modern hyperparameter tuning methods such as Bayesian optimization and Hyperband. It should interface with both Keras and PyTorch.

To answer some of your questions:

1. If you are doing intensive hyperparameter tuning, you need to have three separate datasets: the training set, validation set, and test set. When you train the network, you train on the training set, and evaluate the accuracy metric on the validation set. The validation accuracy is then used to tune the hyperparameters. After you find the best hyperparameters, you then evaluate again on the test set to get the true test accuracy. The reasoning for this is because the act of hyperparameter tuning is itself an optimization procedure, and will suffer from overfitting as well. This is best explained in one of Andrew Ng's video. A more robust way is to do k-fold cross-validation, which I think is the default method of Tune.

2. Yes I think your hyperparameter makes sense. Is $\alpha$ the learning rate? You can also try to include batchsize from some set, say $\{2, 4, 8, 16, ...\}$, given your GPU VRAM.

3. I am not familiar with iRace enough to say it's advantages/disadtanges compared to genetic algorithms. But I know two of the major drawbacks of genetic algorithm is that it doesn't perform well when the number of hyperparameters (or genetic elements) are large, and it has a tedency to converge to local minima.