3
$\begingroup$

As per my knowledge, any entity that is learnable by a training algorithm can be called a parameter. Weights of a neural network are called parameters because of this reason only.

But I have doubts about the qualification of hyperparameter.

Hyperparameter according to my knowledge is an entity that needs to be learned outside of the training algorithm. But, a lot of entities can come into the picture if I want to follow this definition.

For example, the selection of the type of neural network, number of layers, number of neurons in each layer, presence of batch normalization layer, type of activation function, number of parameters, type of parameters (integer, float, etc.), number of epochs, batch size, type of optimizer, learning rate, etc., and I can able to list a lot of entities like this.

Is it okay to call anything that needs to be learned outside the training algorithm a hyperparameter?

$\endgroup$
4
$\begingroup$

In older machine learning literature the given definition of hyperparameters was explicitly the same used in Bayesian statistics, i.e.

a hyperparameter is a parameter of a prior distribution

For example, in Christopher M. Bishop's "Pattern Recognition and Machine Learning" (Springer, 2006), hyperparameters are introduced in the following paragraph (page 30)

Now let us take a step towards a more Bayesian approach and introduce a prior distribution over the polynomial coefficients $\mathbf{w}$. For simplicity, let us consider a Gaussian distribution of the form $$ p(\mathbf{w} \mid \alpha)=\mathcal{N}\left(\mathbf{w} \mid \mathbf{0}, \alpha^{-1} \mathbf{I}\right)=\left(\frac{\alpha}{2 \pi}\right)^{(M+1) / 2} \exp \left\{-\frac{\alpha}{2} \mathbf{w}^{\mathrm{T}} \mathbf{w}\right\} $$ where $\alpha$ is the precision of the distribution, and $M+1$ is the total number of elements in the vector $\mathbf{w}$ for an $M^{\text {th }}$ order polynomial. Variables such as $\alpha$, which control the distribution of model parameters, are called hyperparameters.

In modern machine learning literature though, the definitions became more operational. For example, in Ian Goodfellow, Yoshua Bengio, Aaron Courville - Deep Learning (2016), we can read

Most machine learning algorithms have hyperparameters, settings that we can use to control the algorithm's behavior. The values of hyperparameters are not adapted by the learning algorithm itself (though we can design a nested learning procedure in which one learning algorithm learns the best hyperparameters for another learning algorithm).

So, there is room for interpretation, even though I personally find more technical and precise the old reference to Bayesian statistics. It is clear from that definition that every variable not belonging to the parameters used in the prediction phase but only during training are indeed hyperparameters. Moreover, it is clear that the choice of hyperparameters affects the distribution of learned parameters once the model training reaches convergence.

To elaborate a bit more on the modern definitions, what I don't like about the example taken from Deep Learning is the lack of further explanation about the meaning of "model behavior". Does it refer to weight updating during training? Final metrics score? Both and more? In other words, what are hyperparameters supposed to affect? Of course, these loose definitions do not stop machine learning practitioners from using hyperparameters in the right place, but no surprise about doubts emerging like this question.

$\endgroup$
1
  • $\begingroup$ I would also like to note that the 2 definitions that you quote are not necessarily inconsistent with each other. $\endgroup$
    – nbro
    Oct 29 '21 at 14:51
2
$\begingroup$

Is it okay to call anything that needs to be learned outside the training algorithm a hyperparameter?

I think so, yes.

Personally, I would reserve the term to discuss values that I could choose freely in any given experiment, but that were not learned directly from the training data by the model class I was using. Or perhaps with a tighter restriction for practical purposes, things I would be willing to investigate and change in order to achieve the goal of getting good metrics from the learning agent - it doesn't matter from a practical perspective what I label a value for something that could be changed, if I am not considering changing it.

Choices that I am not interested in making for a given experiment might be hyperparameters to someone else who is performing a different kind of search.

Discrete, top-level choices such as whether to use linear regression versus XGBoost versus a deep neural network might technically be considered hyperparameters, and a sophisticated search algorithm may even automatically test across them. However, I rarely see this kind of choice discussed using the term hyperparameter.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.