In older machine learning literature the given definition of hyperparameters was explicitly the same used in Bayesian statistics, i.e.
a hyperparameter is a parameter of a prior distribution
For example, in Christopher M. Bishop's "Pattern Recognition and Machine Learning" (Springer, 2006), hyperparameters are introduced in the following paragraph (page 30)
Now let us take a step towards a more Bayesian approach and introduce a prior distribution over the polynomial coefficients $\mathbf{w}$. For simplicity, let us consider a Gaussian distribution of the form
$$
p(\mathbf{w} \mid \alpha)=\mathcal{N}\left(\mathbf{w} \mid \mathbf{0}, \alpha^{-1} \mathbf{I}\right)=\left(\frac{\alpha}{2 \pi}\right)^{(M+1) / 2} \exp \left\{-\frac{\alpha}{2} \mathbf{w}^{\mathrm{T}} \mathbf{w}\right\}
$$
where $\alpha$ is the precision of the distribution, and $M+1$ is the total number of elements in the vector $\mathbf{w}$ for an $M^{\text {th }}$ order polynomial. Variables such as $\alpha$, which control the distribution of model parameters, are called hyperparameters.
In modern machine learning literature though, the definitions became more operational. For example, in Ian Goodfellow, Yoshua Bengio, Aaron Courville - Deep Learning (2016), we can read
Most machine learning algorithms have hyperparameters, settings that we can use to control the algorithm's behavior. The values of hyperparameters are not adapted by the learning algorithm itself (though we can design a nested learning procedure in which one learning algorithm learns the best hyperparameters for another learning algorithm).
So, there is room for interpretation, even though I personally find more technical and precise the old reference to Bayesian statistics. It is clear from that definition that every variable not belonging to the parameters used in the prediction phase but only during training are indeed hyperparameters. Moreover, it is clear that the choice of hyperparameters affects the distribution of learned parameters once the model training reaches convergence.
To elaborate a bit more on the modern definitions, what I don't like about the example taken from Deep Learning is the lack of further explanation about the meaning of "model behavior". Does it refer to weight updating during training? Final metrics score? Both and more? In other words, what are hyperparameters supposed to affect? Of course, these loose definitions do not stop machine learning practitioners from using hyperparameters in the right place, but no surprise about doubts emerging like this question.
