The theory behind hyper-parameter optimization (HPO) is not well developed. Nonetheless, there are several hyper-parameter optimization approaches, such as Bayesian optimization (using Gaussian processes), random search, grid search, genetic algorithms, etc. See, for example, the paper Hyperparameter Search in Machine Learning (2015), which attempts to formalize the problem of hyper-parameter optimization in machine learning, Random Search for Hyper-Parameter Optimization (2012), and the related Wikipedia article.
In the paper Hyperparameter Search in Machine Learning (and, similarly, in Random Search for Hyper-Parameter Optimization), the authors formally define the hyper-parameter optimization problem as follows
\begin{align}
\lambda^*
&= \operatorname{arg min}_{\lambda}\mathcal{L}(X^{test}; \mathcal{M} = \mathcal{A}(X^{train}; \lambda)) \tag{1}
\end{align}
where $\lambda$ are the hyper-parameters of the learning algorithm (for example, gradient descent, whose hyper-parameters are the learning rate and the batch size), that is, the algorithm that is used to train the model $\mathcal{M}$ (e.g. a convolutional neural network, with a fixed architecture) using the training ($X^{train}$) and test ($X^{test}$) datasets (for simplicity, ignore cross-validation and related techniques).
In simple words, in equation $1$, we want to find the hyper-parameters $\lambda$ of the learning algorithm $\mathcal{A}$ that minimize the loss $\mathcal{L}$ on the test dataset $X^{test}$, when the model $\mathcal{M}$ is trained using $\mathcal{A}$ and the training dataset $X^{train}$.
The equation $1$ thus ignores the hyper-parameters associated with the model (e.g., the number of layers of a multi-layer perceptron) and only considers the hyper-parameters associated with the learning algorithm. However, note that the optimal hyperparameters of the learning algorithm $\mathcal{A}$ depend on the given training and test datasets, the loss function $\mathcal{L}$ and the model $\mathcal{M}$. Eventually, the formulation in $1$ could be extended to include the hyper-parameters associated with the model (and other hyper-parameters).
So, in general, the choice of the HPO method (including the method you're proposing) depends on several factors, including the model (and its architecture), the task that needs to be solved, the loss function, the training and test datasets, and the computational complexity and runtime efficiency of the HPO method. For example, if the space of hyper-parameters is discrete and small, then grid search (which can be an exhaustive search) will find the best combination of hyper-parameters, for a given task and dataset. However, grid search can be impractical if the search space is huge.
In general, the method you're proposing will not be optimal because, as you state, the hyper-parameters might not be independent of each other. For example, if you're using stochastic gradient descent (that is, you train your model one example at a time), you probably do not want to update the parameters of your model too fast (that is, you probably do not want a high learning rate), given that a single training example is unlikely to be able to give the error signal that is able to update the parameters in the appropriate direction (that is, the global or even local optimum of the loss function). However, if you're using batch gradient descent, the higher the batch size, the more likely you can use a higher learning rate. This example is only meant to give you some intuition, but this might not hold in all cases.
The mostly used hyper-parameter optimization methods (that I mentioned above) seem to assume that the hyper-parameters are, in general, not independent of each other. In fact, this might be a correct assumption, given that, in the real world, the independence assumption almost never holds (see, for example, the related discussions in the context of the naive Bayes classifiers).
