0
$\begingroup$

I am trying to run a regression supervised learning problem. The dataset is not very large and I wanted to do some cross-validation to avoid overfitting. As I have read it's important to do a sensitivity analysis to determine the value of k. Also, I would like to do some hyperparameter grid search for the algorithm (i.e. random forests).

What would be the correct procedure? First take a random value of k and perform the hyperparameter grid search and with the correct hyperparameters do the sensitivity test for k or vice versa?

Thanks in advance!

Improve this question
$\endgroup$
1
  • $\begingroup$ "sensitivity analysis to determine the value of k" What do you mean by this and what is your source? $\endgroup$
    – Ganesh Tata
    Jul 28, 2022 at 3:41

1 Answer 1

Reset to default
0
$\begingroup$

Ideally, the value of k should be fixed and it should not be changed throughout the model training and evaluation process. Here is a good discussion on how to choose the value for k. Once you have done that, you can use k-fold cross validation to evaluate models trained using different hyperparameters.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hey Ganesha, thanks for the contribution. I understand the value of k should be a trade-off between variance and bias. To do so I should iterate through different values of k to find the one that best satisfies both. Would not having the hyperparameters tunned be a problem? $\endgroup$
    – metc
    Jul 30, 2022 at 9:42
  • $\begingroup$ For every fixed k, you should perform hyperparameter tuning, since all your models with different hyperparameters would be trained on the same data and will go through the same evaluation procedure, making it easier to pick the best hyperparameters. $\endgroup$
    – Ganesh Tata
    Jul 30, 2022 at 19:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .