3
$\begingroup$

I'm struggling with calculating accuracy when I do cross-validation for a deep learning model. I have two candidates for doing this. 1. Train a model with 10 different folds and get the best accuracy of them(so I get 10 best accuracies) and average them. 2. Train a model with 10 different folds and get 10 accuracy learning curves. Now, average these learning curves by calculating the mean of 10 accuracies of each epoch. So now we get one averaged accuracy learning curve and find the highest accuracy from this curve.

Among these two candidates which one is correct??

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I guess you could train your model with 10 different folds and in each fold calculate the average accuracy. So you would have 10 values - one corresponding to each fold. And then take the mean of all of them to get the average accuracy of your model.

Your first option doesn't seem great because you take the highest accuracy among folds. If for some reason, the variance between accuracies is high for a fold, this would bias your numbers. Taking mean or maybe median of accuracies might be more reasonable.

Does that help?

Improve this answer
$\endgroup$
2
  • $\begingroup$ But then how should I calculate the average accuracy of each fold? $\endgroup$
    – Juna
    May 13, 2020 at 0:31
  • $\begingroup$ Unless your model has some dynamic component, maybe like RF, I don't think you would need to average for each fold. Actually, you can just evaluate your model once over the model and take that accuracy as the accuracy for the fold. $\endgroup$
    – pecey
    May 14, 2020 at 4:42
0
$\begingroup$

In most cases we choose to take the mean of the k accuracies of k-fold cross validation; that is each time take the one that corresponds to the fold and when every fold has been used as validation set, find the mean accuracy of them.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .