Question is purely theoretical. I am desiging a machine learning model for classification purposes. I am using GridSearch optimization method to select best hyperparameters and I have written separated evolutionary optimizer for selection of proper features used in the model.
My question is how to combine this two optimization tasks, to find the best solution. Two options that comes to my mind are: -Tuning hyperparameters first on all features model, then choose features using that hyperparameters -Selecting best features first on some random hyperparameters and then tune them using previously selected features
However, I believe that these ways might not find the best solution, since for example, some features that worked badly on the used hyperparameters set might work better on a different set, but I will never know, because I will choose the best set for chosen hyperparameters values.
The solution that comes to my mind is to make one big optimisation that will in the same iteration optimize both features selected and the hyperparameters. Or more brutal method, to make "optimization in optimization" meaning that for each iteration of features selection I run seperate grid search of hyperparameters tuning. But I am pretty sure that both solutions are extremely computationally expensive.
What do you think? What is the typical "by the book" way of dealing with these things?
If I didn't explain what I mean clearly enough, please let me know.