1
$\begingroup$

I am trying to determine the result of a cross validation process. Is it just a set of standalone models which is produced after each cross-validation round, or is there some kind of final model which is "aware" of all the models created by the Kfold rounds and benefits from them?

$\endgroup$

1 Answer 1

1
$\begingroup$

The main purpose of a validation procedure is to obtain an accurate estimate of model performance. In brief, in a typical hold out validation, the total dataset is split into a training and test set. A model is then trained on a training set; however, the model's performance (e.g., accuracy) is evaluated on the test set. The idea here is that estimating performance on the same dataset that you used to train the model is obviously going to give you a biased estimate of performance. The performance estimate on the test set is supposed to be a more accurate / unbiased representation of how the model will perform when deployed. However, the randomness of the train-test split may yield spurious results, especially when datasets are small.

This is where cross-validation comes in. Cross-validation repeats this procedure multiple times using different train-test splits on the same dataset. The performance on the different test sets are then averaged over all the iterations giving a more accurate estimate (along with confidence bounds if you calculate the standard deviation) for the model's performance. This averaging approach is much less dependent on a single train-test split, which by chance may not have been representative of the distribution of the overall dataset.

Thus, to answer your question, a cross-validation generates multiple models on (overlapping) subsets of data that are not 'aware" of each other. A k-fold cross-validation, for example, will execute k iterations, generating a new model at each iteration. This scikit learn user guide explains cross-validation (and its many variations) very well.

The cross-validation procedure, as mentioned, is used to get an accurate estimate of model performance in the real world. It's main purpose is not to generate a final model. The final model can be generated by training on the total dataset, but you would not then estimate accuracy by evaluating the performance of that model on the total dataset (as explained this would be a highly biased estimate). Instead, use the average accuracy from the cross-validation.

As an aside, if you decide to save these models after each iteration, you can combine them to make an ensemble model. However, that is not technically a part of the cross-validation procedure.

$\endgroup$
4
  • $\begingroup$ Thank you. Things start getting more clear thanks to your answer. Still, probably misunderstood something previously, not regarding the cross validation but the "simple" validation. Please, correct me if if I am wrong in the following: A-during an epoch the weighs are adjusted each batch as I understand. The adjustion of weights after each batch is based on training set ground truth - nothing to do with validation set here here, right? B-If A is right, than, the validation is being done at the end of each epoch, but not with a goal to adjust weight, but in order to estimate. Right? $\endgroup$
    – Igor
    Jan 8, 2023 at 13:53
  • 1
    $\begingroup$ My pleasure. It seems like you are referring to training of neural networks here by your use of "epoch". The same principles apply. A) and B) are correct. Generally, the data are split into train-validation-test sets for training neural networks [1]. $\endgroup$ Jan 8, 2023 at 16:58
  • 1
    $\begingroup$ After each epoch, the performance of the model is assessed on the validation set. The purpose is to assess for over/underfitting. These articles will help you understand those concepts [1, 2, 3] $\endgroup$ Jan 8, 2023 at 17:04
  • 1
    $\begingroup$ Thank you again. Your explanations are very clear. $\endgroup$
    – Igor
    Jan 8, 2023 at 18:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .