Is it valid to implement hyper-parameter tuning and THEN cross-validation?

Question

I have a multi-label classification task I am solving. I have done hyperparameter tuning (with Keras Tuner) to determine the best configuration for my neural network.

Is it valid to do this (determine the best hyper-parameters) and then do cross-validation to get a more accurate test estimation of the dataset?

I don't see how this would be invalid, given that the cross-validation examples I have seen already have network architectures known a priori, presumably because this is what they chose or feel is the best way of proceeding.

For hyperparameter tuning, all data is split into training and test sets - the training set is further split, when fitting the model, for a 10% validation set - the optimal model is then used to predict on the test set.

For k-fold cross-validation, all data (same as above) is used, but I just split (with sklearn) the data into training and test datasets (so no validation dataset). The test set is used to determine the model performance at each iteration of k-fold cross-validation.

Answer 1

You should not use the training data for hyper-parameter tuning. In other words, when doing the hyper-parameter tuning, you should not optimize the training objective. You should optimize an objective computed on a dataset that is different than the training dataset. This dataset is sometimes called validation dataset, which can also be used for early stopping, which is not a way of hyper-parameter tuning but to avoid over-fitting.

In the Keras Tuner, you can specify the validation data (which is passed to the fit method under the hood) and the objective of the hyper-parameter optimization. You should specify that the objective is computed on the validation data (e.g. val_loss or val_accuracy), which should be different than the training data. Here you have a complete example.

Once you have selected your hyper-parameters, you can train again the best model on a training dataset, which can or not be the same as the training dataset you used during hyper-parameter optimization, but you should use a test dataset to assess the generalization ability of your model. The main requirement is that the training, validation, and test datasets are disjoint in order to avoid bias.

If you use k-fold cross-validation, you will be training and testing your model with different parts of your whole dataset each time. So, if you have $k$ folds, you will use $k - 1$ folds for training and one for testing. You will do this $k$ times, each time with a different fold for testing and the rest for training. This means that you will be using all your data for training and testing, but, each time, the training and test datasets are separate. You use k-fold CV in order to compute an average (and/or variance) estimate of generalisation of your model (you do this especially when your dataset is small).

In principle, I don't see any big problem with doing k-fold cross-validation after you have selected the best hyper-parameters (e.g. architecture), as long as you used a separate validation dataset, which is disjoint from the data you used for k-fold CV. However, make sure to shuffle your data before hyper-parameter tuning (i.e. before selecting the validation dataset), so that the distribution of the training, validation, and test datasets is roughly the same.

If I understand your descriptions correctly, it seems that you used the validation data also during k-fold CV. I don't think this was a good idea, as you're training and testing a model using data that you used to select that model itself (so this can introduce some kind of bias). You also don't specify which metric you optimize during hyper-parameter optimization.