I'm trying to develop skill to deal with very small amount of labeled samples (250 labeled/20000 total, 200 features) practicing on Kaggle "Don't Overfit" dataset (Traget_Practice have provided all 20,000 Targets). I've read a ton of papers and articles on this topic, but nothing what I tried wasn't improved simple regularized SVM result (best acc 75/auc 85) or any other algorithm result (LR, K-NN, NaiveBayes, RF, MLP). I believe that result can be better (on Leaderboard they go even over auc 95)
What I've tried without success:
Remove outliers I've tried to remove 5%-10% outliers with EllipticEnvelope and with IsolationForest.
Feature Selection I've tried RFE (with or without CV) + L1/L2 regularised LogisticRegression, and SelectKBest (with chi2).
- Semi-Supervised techniques I've tried Co-training with different combinations of two complementary algorithms and :100-100: splitted features. Also I've tried LabelSpreading, but I don't know how to provide most uncertain samples (I tried predictions from another algorithms but there was a lot of mislabeled samples and without any success).
- Ensembling Classifiers StackingClassifier with all possible combinations of algorithms and this also doesn't improve result (best is same as SVM acc75/auc 85).
Can anyone give me advice what I'm doing wrong or what else to try?