How to deal with an unbalanced dataset?

Question

I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a stroke (1) and 4861 that did not (0). The accuracy is (as a result of this, I suppose) very high (89% on the first epoch, and 95% on the second, then it stays at 95%). What would be the best thing to do about this?

Answer 1

4861/5111 is about 95.1%, so it looks like your classifier is probably predicting every patient as "no stroke" (i.e. it is not really doing anything useful). The thing to do is to work out the costs of false-positive (predicting a stroke but it didn't happen) and false-negative (predicting they won't have a stroke but they did) errors. Then factor those into your training process. This can be done in two ways: Firstly by using a weighted training criterion so you can factor the costs in explicitly; or secondly by resampling the dataset to have a greater proportion of "stroke" patients.

The presence of an imbalance is very rarely a justification in itself for resampling the data. The real reason to do so is that tasks with an imbalance very often have unequal misclassification costs, for instance if you tell someone they are going to have a stroke when they won't, you will have scared them rather badly, and you may spend some more money on more testing, but that is probably about it. If you tell someone they are not at risk of a stroke when they are, they may go home, have a stroke and become severely disabled as a result, or even die. So the false-negative cost is likely to be much higher and that will boost recognition of the positive cases. The amount of resampling required depends only on the costs; the degree of imbalance is entirely irrelevant.

BTW, rather than look at accuracy, you might want to look at a related metric, which is the improvement over a classifier that always predicts the majority class. Something like:

$$\frac{\mathrm{Accuracy} - \pi}{1 - \pi}$$

where $\pi$ is the proportion of the most common class in the dataset. In this case, your classifier is going to score somewhere close to zero as it is probably just going by the majority class, and a score of zero shows clearly that it isn't doing a good job. A score of 1 would be perfect classification. A negative score shows the model is worse than just guessing. It is an affine transformation of accuracy though, so it is still measuring the same basic thing, just on a more interpretable scale.

Answer 2

You can use a data augmentation technique like SMOTE to oversample the minority class. It will help you have a more balanced dataset. Here is a nice guide on it: https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/

Answer 3

You should use another classification metric for evaluating your model. I would just look at the confusion matrix to see how the model performance on the "interesting class" (minority class).

To overcome your imbalanced dataset, you could upsample the minority class. Bootstrapping is a great staring point. Then advance with SMOTE or something else. This article might give you some ides; 8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset