2
$\begingroup$

I am working on a credit card fraud detection problem using autoencoders. I have a question regarding the dataset I'll be using.

I've downloaded the dataset for the above problem from Kaggle, which is highly imbalanced: it contains only 492 frauds out of 284,807 transactions. Why isn't the dataset already balanced? Should I balance it before applying the auto-encoder?

$\endgroup$
  • $\begingroup$ I've edited your post to leave only one question because, otherwise, your post should have been closed as too broad. Furthermore, I've added a link to the supposed dataset you are talking about and I've updated the info about this dataset. $\endgroup$ – nbro Dec 26 '19 at 15:40
2
$\begingroup$

I believe that the idea is to have a similar ratio of fraud/"normal transaction" as to the ones that bank encounter on real life.

If you balance it you will probably have a lot of false positive once you apply your solution to real world's data and, if that may be fine for you to play with, it's not what a bank would like as they can't block too much of the "normal" transaction or the client will change of bank. According to this post (https://www.quora.com/How-many-transactions-do-typical-banks-process-everyday) just the mastercard alone represent 3.4billion transaction/day, just imagine if 1% of the daily transaction where blocked every day, it would represent 34 million of transactions blocked without any valid reason.

It's different from a lot of classification problem where you want to have balanced dataset, here you try to detect anomalies and, by definition, they are rare so they should be as rare in your dataset.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.