4
$\begingroup$

I am working on an anti-fraud project. In the project, we are trying to predict the fraud user in the out time data set. But the fraud user has a very low ratio, only 3%. We expect a model with a precision more than 15%.

I tried Logistic Regression, GBDT+LR, xgboost. All models are not good enough. Step wise Logistic Regression performs best, which has a precision of 9% with recall rate 6%.

Is there any other models that I can use for this problem or any other advise ?

$\endgroup$
  • 4
    $\begingroup$ Welcome to ai.se..This is a special class of problem called anomaly detection problem..Just do a quick Google search Andrew ng Coursera anomaly detection. $\endgroup$ – DuttaA Apr 19 '18 at 6:24
  • $\begingroup$ @DuttaA This should probably be upgraded to an answer, since the OP's problem is exactly what Anomaly Detection is for. $\endgroup$ – DrMcCleod Apr 22 '18 at 8:32
1
$\begingroup$

You can balance your data-set.

Many models work with batches of samples. If you have a very unbalanced dataset, you can simply split it and ensure your batches are balanced (for example, for a Neural Network, using minibatches of 32 samples, you could draw 16 from your fraud users, and 16 from non-fraud users).

During the learning phase, this ensures the model doesn't just output the most common class, but instead tries to learn to distinguish both.

$\endgroup$
  • $\begingroup$ Are you sure about that? Because the less the number of samples poorer the generalization $\endgroup$ – DuttaA Apr 19 '18 at 19:31
  • $\begingroup$ Ive used it before for a skewed movie ratings dataset without issues. Keep in mind ypu still use the same amount of samples. You just change their relative frequency, such that both classes are equally likely to appear $\endgroup$ – BlueMoon93 Apr 19 '18 at 23:21
0
$\begingroup$

Heavily imbalanced classification tasks do not need a certain type of model, you can get different ones to work.

You have two options: either use class weights (for example setting them to 'balanced' in the ScikitLearn SVM) in order to indicate that samples from a class are more important (the underrepresented one) or rebalance your dataset. For rebalancing purposes, and assuming you are using Python, I recommend Imbalanced Learn. There you have algorithms for over-sampling, under-sampling, over-sampling followed by under-sampling or ensemble sampling. If you use them, please check the plausibility of the synthetic samples you created by reducing dimensionality first and then plotting them for example in two dimensions. Are the synthetic samples similar to the true class?

I would also recommend you to think about relevant metrics for (heavily) imbalanced problems and consider the no-information rate. That is another question though.

$\endgroup$
  • $\begingroup$ I tried under-sampling. It performs quite unstable between each run, some times better and most of the time it performs worse. $\endgroup$ – BerSerK Apr 24 '18 at 9:45
  • $\begingroup$ What I would do is not try one of the techniques randomly, but reduce your data to 2D and plot it before and after the transformation. Then check for plausibility. Try also for example the Smote-Tomek technique, which combines Over- and Undersampling. If both methods do not work to your satisfaction simply use class weights. $\endgroup$ – AlexGuevara Apr 24 '18 at 11:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.