10
$\begingroup$

I'm trying to develop skills to deal with very small amounts of labeled samples (250 labeled/20000 total, 200 features) by practicing on Kaggle "Don't Overfit" dataset (Traget_Practice have provided all 20,000 Targets). I've read a ton of papers and articles on this topic, but everything I've tried did not improve simple regularized SVM results (best accuracy was 75 and AUC was 85) or any other algorithm result (LR, K-NN, NaiveBayes, RF, MLP). I believe the result can be better (on Leaderboard they go even over AUC 95)

What I've tried without success:

  • Remove outliers I've tried to remove 5%-10% outliers with EllipticEnvelope and with IsolationForest.

  • Feature Selection I've tried RFE (with or without CV) + L1/L2 regularised LogisticRegression, and SelectKBest (with chi2).

  • Semi-Supervised techniques I've tried co-training with different combinations of two complementary algorithms and :100-100: split features. I've also tried LabelSpreading, but I don't know how to provide the most uncertain samples (I tried predictions from other algorithms, but there were many mislabeled samples, hence was unsuccessful).

  • Ensembling Classifiers StackingClassifier with all possible combinations of algorithms and this also didn't improve the result (the best is the same as SVM accuracy 75 and AUC 85).

Can anyone give me advice on what I'm doing wrong or what else to try?

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Would it be easy to hand-label each data point? If so, you might try active learning. $\endgroup$
    – Philip Raeisghasem
    May 4, 2019 at 22:07
  • $\begingroup$ I know for that technique (Label Spreading work similar - only asking for most uncertain samples). But no, there is 200 continuous features which we don't know what they mean, so it is impossible to label them by hand. $\endgroup$
    – FirePower
    May 6, 2019 at 11:08
  • 1
    $\begingroup$ In the forums on the competition they go into several of the approaches, i reccomend looking there $\endgroup$
    – mshlis
    Jul 3, 2019 at 12:06
  • $\begingroup$ Just as an idea: Maybe you can try a neural network and initialize it with parameters returned from an autoencoder. This way the autoencoder can learn useful representations and you can start from there $\endgroup$
    – amin
    Oct 25, 2020 at 12:17
  • $\begingroup$ Take a look at seme-supervised learning (with GAN). For example, this paper or that one. Also, here is my results. Let my know, if need more info on it, I'll post then an answer $\endgroup$
    – Aray Karjauv
    Oct 25, 2020 at 15:56

1 Answer 1

Reset to default
1
$\begingroup$

In that particular competition, you can try using GAN to generate new data or adding noise to existing data. You can also use K-means algorithm. You can try using a smaller network and remove bias. May be you can use logistic regression to compare the result. You can also use a PCA method.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Upsampling (adding noise to labeled samples which results in new samples) can be an effective practice. However, I'm not sure about the rest of the above answer. For example, I'm afraid 250 labeled samples may not be enough for a GAN as we have 200 feature columns. Or, according to the question, the best result is achieved by SVM (not an NN algorithm) so I'm not sure what "using a smaller network and remove bias" means in this context. It would be great if you can provide more details about the applications of K-Means and PCA here. $\endgroup$
    – Borhan Kazimipour
    Jun 4, 2019 at 6:30
  • $\begingroup$ May be add some noise to the data to produce more $\endgroup$
    – Clement
    Jun 4, 2019 at 6:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .