Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I am working on a credit card fraud detection problem using autoencoders. I have a question regarding the dataset I'll be using.
I've downloaded the dataset for the above problem from Kaggle, which is highly imbalanced: it contains only 492 frauds out of 284,807 transactions. Why isn't the dataset already balanced? Should I balance it before applying the auto-encoder?
I believe that the idea is to have a similar ratio of fraud/"normal transaction" as to the ones that bank encounter on real life.
If you balance it you will probably have a lot of false positive once you apply your solution to real world's data and, if that may be fine for you to play with, it's not what a bank would like as they can't block too much of the "normal" transaction or the client will change of bank. According to this post (https://www.quora.com/How-many-transactions-do-typical-banks-process-everyday) just the mastercard alone represent 3.4billion transaction/day, just imagine if 1% of the daily transaction where blocked every day, it would represent 34 million of transactions blocked without any valid reason.
It's different from a lot of classification problem where you want to have balanced dataset, here you try to detect anomalies and, by definition, they are rare so they should be as rare in your dataset.
