According this thread some hyperparameters are independent from each other while some are directly related.
One of the answers give an example where two hyperparameters affect each other.
For example, if you're using stochastic gradient descent (that is, you train your model one example at a time), you probably do not want to update the parameters of your model too fast (that is, you probably do not want a high learning rate), given that a single training example is unlikely to be able to give the error signal that is able to update the parameters in the appropriate direction (that is, the global or even local optimum of the loss function).
How would someone creating a neural network know how the hyperparameters affect each other?
In other words, what are the heuristics for hyperparameter selection when trying to build a robust model?