I'm trying to develop skill to deal with very small amount of labeled samples (250 labeled/20000 total, 200 features) practicing on Kaggle "Don't Overfit" dataset (Traget_Practice have provided all 20,000 Targets). I've read a ton of papers and articles on this topic, but nothing what I tried wasn't improved simple regularized SVM result (best acc 75/auc 85) or any other algorithm result (LR, K-NN, NaiveBayes, RF, MLP). I believe that result can be better (on Leaderboard they go even over auc 95)

What I've tried without success:

  • Remove outliers I've tried to remove 5%-10% outliers with EllipticEnvelope and with IsolationForest.

  • Feature Selection I've tried RFE (with or without CV) + L1/L2 regularised LogisticRegression, and SelectKBest (with chi2).

  • Semi-Supervised techniques I've tried Co-training with different combinations of two complementary algorithms and :100-100: splitted features. Also I've tried LabelSpreading, but I don't know how to provide most uncertain samples (I tried predictions from another algorithms but there was a lot of mislabeled samples and without any success).
  • Ensembling Classifiers StackingClassifier with all possible combinations of algorithms and this also doesn't improve result (best is same as SVM acc75/auc 85).

Can anyone give me advice what I'm doing wrong or what else to try?

  • 1
    $\begingroup$ Would it be easy to hand-label each data point? If so, you might try active learning. $\endgroup$ – Philip Raeisghasem May 4 '19 at 22:07
  • $\begingroup$ I know for that technique (Label Spreading work similar - only asking for most uncertain samples). But no, there is 200 continuous features which we don't know what they mean, so it is impossible to label them by hand. $\endgroup$ – FirePower May 6 '19 at 11:08
  • $\begingroup$ In the forums on the competition they go into several of the approaches, i reccomend looking there $\endgroup$ – mshlis Jul 3 '19 at 12:06

In that particular competition, you can try using GAN to generate new data or adding noise to existing data. You can also use K-means algorithm. You can try using a smaller network and remove bias. May be you can use logistic regression to compare the result. You can also use a PCA method.

| improve this answer | |
  • $\begingroup$ Upsampling (adding noise to labeled samples which results in new samples) can be an effective practice. However, I'm not sure about the rest of the above answer. For example, I'm afraid 250 labeled samples may not be enough for a GAN as we have 200 feature columns. Or, according to the question, the best result is achieved by SVM (not an NN algorithm) so I'm not sure what "using a smaller network and remove bias" means in this context. It would be great if you can provide more details about the applications of K-Means and PCA here. $\endgroup$ – Borhan Kazimipour Jun 4 '19 at 6:30
  • $\begingroup$ May be add some noise to the data to produce more $\endgroup$ – Clement Hui Jun 4 '19 at 6:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.