1
$\begingroup$

I am dealing with an intent classification task on an Italian customer service data set.

I've more or less 1.5k sentences and 29 classes (imbalanced).

According to the literature, a good choice is to generate synthetic data, oversampling, or undersampling the training data, using for example the SMOTE algorithm.

I also want to use a cross-validation mechanism (stratified k-fold) to be more confident in the obtained result.

I also know that accuracy is not the right metric to take into account, I should use precision, recall, and confusion matrix.

Is it possible to combine k-fold cross-validation and oversampling (or undersampling) techniques?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

It is straightforward to combine k-fold cross-validation with a technique like oversampling or undersampling.

First, apply the balance-restoration technique to your training data. Then parametrize a model using k-fold cross-validation on the re-balanced training data. In Scikit learn, I believe you can even bundle these actions together into a single 'pipeline' object to make it easier to manipulate.

Precision/recall is probably a fine starting place for measuring performance.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So you mean first divide data into train and test set (hold out), rebalance train with Smote (for example) and then apply cross validation on this train set. What about instead, do oversample technique only during k fold iteration? let me explain better. Suppose i have unbalanced train, i apply 10-fold cross validation, so 9 groups will be used as train and 1 as test, what i mean is to apply oversample only at this time on the 9 train groups, just to reduce overfitting maybe. I hope i have been clear. i will wait your advices. thanks $\endgroup$
    – Alfonso
    Mar 12, 2019 at 21:12
  • $\begingroup$ @Alfonso You're quite right. That would be a better idea. My protocol leaks information from the validation set and yours does not. $\endgroup$
    – John Doucette
    Mar 12, 2019 at 21:14
  • $\begingroup$ thank you for your valuable advice @John Doucette $\endgroup$
    – Alfonso
    Mar 12, 2019 at 21:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .