4
$\begingroup$

I have a very imbalanced dataset of two classes: 2% for the first class and 98% for the second. Such imbalance does not make training easy and so balancing the data set by undersampling class 2 seemed like a good idea.

However, as I think about it, should not the machine learning algorithm expect the same data distribution in nature as in its training set? I know, for sure, that the distribution of data in nature matches my imbalanced dataset. Does that mean that the balanced dataset will negatively affect the neural net performance with testing? when it assumed a different distribution of data caused by my balanced data set.

$\endgroup$
3
$\begingroup$

This is a very good question. Your problem is the classic classification problem of Neural Networks. In this problem the main objective of the Neural Network is to transform the data by some non-linear (in general) transformation so that the data becomes linearly separable for the final layer to perform classification.

Point to Note: This is not a regression problem, so that you are trying to fit a curve. Whenever there is a regression problem logically you can use a PDF to write some kind of information about new data. You can express mathematically the probability of your data falling within a certain range of error, since this is a optimising continuous function problem (generally RMSE).

This is not in the case of classifier. Classifier follow Bernoulli probability (though we represent the cost function as continuous). So current event is independent of past events. This makes a classifier harder to train for unbalanced class. So if we write:

func foo(data):
    return True

It pretty much has 98% accuracy, but you can understand we do not want this type of classifier.

In general we want both classes to have good accuracy scores, sometimes this is measured by $F1_{score}$ but I like to think it in terms of proportionality. If we have $n$ examples with $m$ in one class, then and the classifier $a_1$ and $a_2$ correct predictions in both classes respectively, then I would check both the metric $ \frac {a_1}{m}$ and $ \frac {a_2}{n-m}$, which gives you the general idea.

Also in practical the identification 2% is sometimes far more important than the rest 98% (air-plane defects, cancer detection). So we use special ML algorithm called Anomaly Detectors for such type of problems.

$\endgroup$
  • $\begingroup$ Side note: I have tried to use mathematical terms to convey my general idea. Due to my unfamiliarity with mathematical terms the interpretation might turn out to be wrong. Feel free to point out errors or inconsistencies. $\endgroup$ – DuttaA Sep 17 '18 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.