I am dealing with an intent classification task on an Italian customer service data set.

I've more or less 1.5k sentences and 29 classes (imbalanced).

According to the literature, a good choice is to generate synthetic data, oversampling, or undersampling the training data, using for example the SMOTE algorithm.

I also want to use a cross-validation mechanism (stratified k-fold) to be more confident in the obtained result.

I also know that accuracy is not the right metric to take into account, I should use precision, recall, and confusion matrix.

Is it possible to combine k-fold cross-validation and oversampling (or undersampling) techniques?


It is straightforward to combine k-fold cross-validation with a technique like oversampling or undersampling.

First, apply the balance-restoration technique to your training data. Then parametrize a model using k-fold cross-validation on the re-balanced training data. In Scikit learn, I believe you can even bundle these actions together into a single 'pipeline' object to make it easier to manipulate.

Precision/recall is probably a fine starting place for measuring performance.

  • $\begingroup$ So you mean first divide data into train and test set (hold out), rebalance train with Smote (for example) and then apply cross validation on this train set. What about instead, do oversample technique only during k fold iteration? let me explain better. Suppose i have unbalanced train, i apply 10-fold cross validation, so 9 groups will be used as train and 1 as test, what i mean is to apply oversample only at this time on the 9 train groups, just to reduce overfitting maybe. I hope i have been clear. i will wait your advices. thanks $\endgroup$ – Alfonso Mar 12 '19 at 21:12
  • $\begingroup$ @Alfonso You're quite right. That would be a better idea. My protocol leaks information from the validation set and yours does not. $\endgroup$ – John Doucette Mar 12 '19 at 21:14
  • $\begingroup$ thank you for your valuable advice @John Doucette $\endgroup$ – Alfonso Mar 12 '19 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.