# Does this hyperparameter optimisation approach yield the optimal hyperparameters?

Say I have a ML model which is not very costly to train. It has around say 5 hyperparameters.

One way to select best hyperparameters would be to keep all the other hyperparamaters fixed and train the model by changing only one hyperparameter within a certain range. For the sake of mathematical convenience, we assume for the hyperparameter $$h^1$$, keeping all other hyperparameters fixed to their initial values, the model performs best when $$h^1_{low} < h^1 < h^1_{high}$$ (which we found out by running the model on a huge range of $$h^1$$). Now we, fix $$h^1$$ to one of the best values and tune $$h^2$$ the same way, where $$h^1$$ is chosen and the rest of the hyperparameters are again fixed on their initial values.

My question is: Does this method find the best hyperparameter choices for the model? I know if the hyperparameters are independent, then this definitely does find the best solution, but in a general case, what is the general theory around this? (NOTE: I am not asking about the problem of choosing hyperparamaters, but I am asking about the aforementioned approach of choosing hyperparameters)

After you've computed $$h^{1}_{optimal}$$ the only thing you can be sure is that this is the best (assuming constrained case) value of $$h^1$$ (with respect to some model performance metric) given your initial values for $$h^2, ..., h^n$$. If you change a bit any of $$h^2, ..., h^n$$ you're no longer certain that the value $$h^1$$ you found is the optimal one. So yes, the key here is the assumption about the independence.

The theory behind hyper-parameter optimization (HPO) is not well developed. Nonetheless, there are several hyper-parameter optimization approaches, such as Bayesian optimization (using Gaussian processes), random search, grid search, genetic algorithms, etc. See, for example, the paper Hyperparameter Search in Machine Learning (2015), which attempts to formalize the problem of hyper-parameter optimization in machine learning, Random Search for Hyper-Parameter Optimization (2012), and the related Wikipedia article.

In the paper Hyperparameter Search in Machine Learning (and, similarly, in Random Search for Hyper-Parameter Optimization), the authors formally define the hyper-parameter optimization problem as follows

\begin{align} \lambda^* &= \operatorname{arg min}_{\lambda}\mathcal{L}(X^{test}; \mathcal{M} = \mathcal{A}(X^{train}; \lambda)) \tag{1} \end{align}

where $$\lambda$$ are the hyper-parameters of the learning algorithm (for example, gradient descent, whose hyper-parameters are the learning rate and the batch size), that is, the algorithm that is used to train the model $$\mathcal{M}$$ (e.g. a convolutional neural network, with a fixed architecture) using the training ($$X^{train}$$) and test ($$X^{test}$$) datasets (for simplicity, ignore cross-validation and related techniques).

In simple words, in equation $$1$$, we want to find the hyper-parameters $$\lambda$$ of the learning algorithm $$\mathcal{A}$$ that minimize the loss $$\mathcal{L}$$ on the test dataset $$X^{test}$$, when the model $$\mathcal{M}$$ is trained using $$\mathcal{A}$$ and the training dataset $$X^{train}$$.

The equation $$1$$ thus ignores the hyper-parameters associated with the model (e.g., the number of layers of a multi-layer perceptron) and only considers the hyper-parameters associated with the learning algorithm. However, note that the optimal hyperparameters of the learning algorithm $$\mathcal{A}$$ depend on the given training and test datasets, the loss function $$\mathcal{L}$$ and the model $$\mathcal{M}$$. Eventually, the formulation in $$1$$ could be extended to include the hyper-parameters associated with the model (and other hyper-parameters).

So, in general, the choice of the HPO method (including the method you're proposing) depends on several factors, including the model (and its architecture), the task that needs to be solved, the loss function, the training and test datasets, and the computational complexity and runtime efficiency of the HPO method. For example, if the space of hyper-parameters is discrete and small, then grid search (which can be an exhaustive search) will find the best combination of hyper-parameters, for a given task and dataset. However, grid search can be impractical if the search space is huge.

In general, the method you're proposing will not be optimal because, as you state, the hyper-parameters might not be independent of each other. For example, if you're using stochastic gradient descent (that is, you train your model one example at a time), you probably do not want to update the parameters of your model too fast (that is, you probably do not want a high learning rate), given that a single training example is unlikely to be able to give the error signal that is able to update the parameters in the appropriate direction (that is, the global or even local optimum of the loss function). However, if you're using batch gradient descent, the higher the batch size, the more likely you can use a higher learning rate. This example is only meant to give you some intuition, but this might not hold in all cases.

The mostly used hyper-parameter optimization methods (that I mentioned above) seem to assume that the hyper-parameters are, in general, not independent of each other. In fact, this might be a correct assumption, given that, in the real world, the independence assumption almost never holds (see, for example, the related discussions in the context of the naive Bayes classifiers).