Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
I am trying to run a regression supervised learning problem. The dataset is not very large and I wanted to do some cross-validation to avoid overfitting. As I have read it's important to do a sensitivity analysis to determine the value of k. Also, I would like to do some hyperparameter grid search for the algorithm (i.e. random forests).
What would be the correct procedure? First take a random value of k and perform the hyperparameter grid search and with the correct hyperparameters do the sensitivity test for k or vice versa?
Ideally, the value of k should be fixed and it should not be changed throughout the model training and evaluation process. Here is a good discussion on how to choose the value for k. Once you have done that, you can use k-fold cross validation to evaluate models trained using different hyperparameters.