4
$\begingroup$

I want to prevent my model from overfitting. I think that k-fold cross-validation (because it is doing this each time with different datasets) may be more effective than splitting the dataset into training and test datasets to prevent overfitting, but a colleague (who has little experience in ML) says that, to prevent overfitting, the 70/30% split performs better than the cross-validation. In my opinion, k-fold cross-validation provides a reliable method to test the model performance.

Is k-fold cross-validation more effective than splitting the dataset into training and test datasets to prevent overfitting? I am not concerned with computational resources.

$\endgroup$
  • $\begingroup$ Just use k-fold cross validation, time consuming but the best results. $\endgroup$ – DuttaA Nov 17 at 11:04
  • 2
    $\begingroup$ The question is missing some details. Cross validation is usually done on some data split, and 30% is high but not unreasonable in that case, so it is not clear what is actually different between your approach and that proposed by your colleagues. Are you referring to yourself doing k-fold cross-validation, or that you will perform 3-way split train/cv/test whilst colleague just suggests train/test? Please clarify using edit to update the question $\endgroup$ – Neil Slater Nov 17 at 15:42
  • $\begingroup$ @DuttaA I thought the purpose of cross validation was to have a better evaluation of the model on data. What do you mean by best results? If it is overfitting, it will still be overfitting after cross validation, only you will know it for sure how much it is overfitting. Please let me know if I am missing something here. $\endgroup$ – serali Nov 18 at 14:06
  • 1
    $\begingroup$ @DuttaA Isn't it the purpose of train-test-validation split to check for overfitting? On top of that you can do cross validation to get a mean and standard deviation of accuracy(or any metric of choice) on your dataset to have a statistical summary of your result. Overfitting will roughly be the same in all runs of a cross validation; unless the dataset is horribly unbalanced or the splitting is not done right. $\endgroup$ – serali Nov 18 at 14:53
  • 1
    $\begingroup$ @serali Cross-validation can also be used in the case you have a small training dataset. In those cases, you will train your model on all subsets of the dataset. $\endgroup$ – nbro Nov 18 at 15:24
3
$\begingroup$

K-fold cross-validation is probably preferred in terms of completeness and generalization: you ensure that the system has seen the complete dataset for training. However, in deep learning this is often not feasible due to time and power constraints. They can both be used, and there is not one better than the other. It really depends on the specific case, the size of the dataset and the time and hardware available. Note that overfitting can be (partially) remedied by things such as dropout.

To be fair: it is fine to have a discussion about this with your colleagues, but as so often there is no one correct answer. If you really want proof, you can test it out and compare them. But performance-wise (i.e. the model's predictive power), the difference will be small.

$\endgroup$
2
$\begingroup$

Purely in terms of overfitting, and assuming you train both for equal amounts of time, 70/30 is probably better but performance is not going to be very good. Not training on %30 of data will make both training and test results equally bad (in my opinion). But it won't overfit, that is for sure. Cross validation (you have in mind 90/10, I assume) will take a long time, so that won't have enough time to train and it might be overfitting more compared to 70/30, but as it is going to see all training samples %90 at a time, there is a good chance it will train better. So, at the end of the day, it will overfit more but perform better.

If you are asking which is better overall, performance and overfitting, I say it depends on the size of your dataset. If you have millions of samples in it, you can even use a 98/1/1 for training, testing and validation and still be OK.

Edit: Thinking a little more about it, even if the time is not an issue the situation will roughly be the same. But you will know the performance of the model on new data to a higher certainty with cross validation.

$\endgroup$
  • 1
    $\begingroup$ What do you mean with: ",it will overfit a more but perform better"? $\endgroup$ – jennifer ruurs Nov 17 at 11:14
  • 1
    $\begingroup$ Using loss as a criteria, I mean both training and test results will be higher, but the ratio of test loss to training loss will be lower compared to 70/30 split. $\endgroup$ – serali Nov 17 at 11:19
  • 1
    $\begingroup$ I meant both training and test results will be lower, not higher in above comment. And also training loss to test loss ratio. It is loss, not accuracy so the lower the better. I am unable to edit the comment, sorry for the second comment. $\endgroup$ – serali Nov 17 at 11:41
  • $\begingroup$ No i understood what you are trying to say, thank you $\endgroup$ – jennifer ruurs Nov 17 at 11:44
0
$\begingroup$

Both methods are fine if used properly. As a rule of thumb, when training time is not an issue, use split method if you have more data than you can use in your model and cross-validation if not. I would suggest handling overfitting by some other means.

$\endgroup$
  • $\begingroup$ Jeesus christ, nbro. "Used properly" in the sense that of course you mess any method if you don't know what you are doing. So better to use a simpler method (split) than more complicated which you don't understand. OP did not ask about how to deal with overfitting so I'm not going to suggest any means for that. Rule of thumb is a rule of thumb. Results may vary. But split lets you have less data available for training than cross validation so that's why cv would generally be better all other things being equal. $\endgroup$ – tjk Nov 18 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.