0
$\begingroup$

I'm constructing a feed forward neural network that predicts whether a patient will get a stroke or not. However, my dataset is very unbalanced. Out of 5111 rows, 250 contain patients that have had a stroke (1) and 4861 that did not (0). The accuracy is (as a result of this, I suppose) very high (89% on the first epoch, and 95% on the second, then it stays at 95%). What would be the best thing to do about this?

$\endgroup$
1
  • 1
    $\begingroup$ Please, put your specific question in the title. "Unbalanced dataset" is not specific and not a question. Moreover, use more appropriate tags, like unbalanced-datasets. Finally, I recommend that you take a look at existing questions. I think this question was already asked more than once here in the past. $\endgroup$
    – nbro
    Apr 28 at 14:26

3 Answers 3

1
$\begingroup$

4861/5111 is about 95.1%, so it looks like your classifier is probably predicting every patient as "no stroke" (i.e. it is not really doing anything useful). The thing to do is to work out the costs of false-positive (predicting a stroke but it didn't happen) and false-negative (predicting they won't have a stroke but they did) errors. Then factor those into your training process. This can be done in two ways: Firstly by using a weighted training criterion so you can factor the costs in explicitly; or secondly by resampling the dataset to have a greater proportion of "stroke" patients.

The presence of an imbalance is very rarely a justification in itself for resampling the data. The real reason to do so is that tasks with an imbalance very often have unequal misclassification costs, for instance if you tell someone they are going to have a stroke when they won't, you will have scared them rather badly, and you may spend some more money on more testing, but that is probably about it. If you tell someone they are not at risk of a stroke when they are, they may go home, have a stroke and become severely disabled as a result, or even die. So the false-negative cost is likely to be much higher and that will boost recognition of the positive cases. The amount of resampling required depends only on the costs; the degree of imbalance is entirely irrelevant.

BTW, rather than look at accuracy, you might want to look at a related metric, which is the improvement over a classifier that always predicts the majority class. Something like:

$$\frac{\mathrm{Accuracy} - \pi}{1 - \pi}$$

where $\pi$ is the proportion of the most common class in the dataset. In this case, your classifier is going to score somewhere close to zero as it is probably just going by the majority class, and a score of zero shows clearly that it isn't doing a good job. A score of 1 would be perfect classification. A negative score shows the model is worse than just guessing. It is an affine transformation of accuracy though, so it is still measuring the same basic thing, just on a more interpretable scale.

$\endgroup$
0
$\begingroup$

You can use a data augmentation technique like SMOTE to oversample the minority class. It will help you have a more balanced dataset. Here is a nice guide on it: https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/

$\endgroup$
1
  • 1
    $\begingroup$ I'd advise against SMOTE for modern classifier systems. It was developed for very simple classifiers that didn't support cost-sensitive learning or have some means of avoiding over-fitting. Modern classifier systems, such as SVMs or NN have both, so SMOTE is not likely to be necessary (or helpful as there is a lot of good theory supporting regularisation, but none for the rather peculiar method SMOTE uses to generate synthetic examples) $\endgroup$ Apr 29 at 9:21
0
$\begingroup$

You should use another classification metric for evaluating your model. I would just look at the confusion matrix to see how the model performance on the "interesting class" (minority class).

To overcome your imbalanced dataset, you could upsample the minority class. Bootstrapping is a great staring point. Then advance with SMOTE or something else. This article might give you some ides; 8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset

$\endgroup$
1
  • 1
    $\begingroup$ The classification metric should depend on the needs of the application rather than problems with the model. In this application the expected loss is likely to be a better metric as the misclassification costs are obviously unequal, so the thing to do is to work out what those costs reasonably might be. $\endgroup$ Apr 29 at 9:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .