I have a fully connected feedforward classifier neural network that uses the leaky ReLU activation function. I would like to apply a state-of-the-art hyperparameter tuning method to my methodology. Currently, the fitness function associated with each parameter setting is a weighted combination of (i) accuracy of both training and test data sets ($acc_{tr}$ and $acc_{ts}$, respectively), (ii) numbers of activated hidden layers ($nL$) and their neurons ($nN_l$), and (iii) number of epochs ($nE$) - length of training. The fitness function is computed as follows:
\begin{equation}\label{eqq} FF = 0.7 \times (acc_{tr} + acc_{ts}) + 0.1 \times \frac{nL}{10} + 0.1 \times \frac{\sum_{l \in \mathcal{L}} nN_l}{1000} + 0.1 \times \frac{nE}{2000} \end{equation}
Note that I have determined the weights in function $FF$ based on my experimental observations. Potential values for the parameters of this neural network are given in the following table:
$\alpha$ here is the leakage parameter of the ReLU activation function. Considering that I am new to this field, my questions are as follows:
- Is the weighted fitness function an appropriate one? Should I include both training and test accuracies at the same time? Why?
- Do the potential values make sense for a fully connected feedforward classifier neural network with the leaky ReLU activation function? Are there any other parameters that I have to tune? Please consider that I have developed this neural network using the Keras library.
- The above table also shows the elite configuration. I have found this parameter setting using the iRace package. Do you recommend any other method for hyperparameter tuning in neural networks?
- I have seen many papers use genetic algorithms to tune their parameters. What are the advantages of using a genetic algorithm in comparison to the iRace package (if there is any)?
Please kindly answer my above questions with supporting scientific references. Also, please let me know if there are any other considerations for the hyperparameter tuning of neural networks. Thank you!