My dataset has essentially multi-classification problem, where I have the treatment failure (0), cure (1) and relapses (3) of patients that are associated with a series of covariates (~100 different features, SEX, BMI, HIV, RNA Signatures etc.). In this dataset, I have data 100 patients and 80 are considered cured, 10 failure and 10 relapse.

Is there a problem that I only have little data for the treatment failure? How accurate would my treatment failures predictions be if my I have so little data to train and test it?


1 Answer 1


You are right. This is a broad question, but I'll try to provide some general direction. Let's go through the whole pipeline from data collection to model selection, training, and evaluation. These are just basic things to consider. Choose and experiment based on your dataset. With limited data, it will be tough to generalize, but here is checklist:

1. Data collection: I'm not sure if it’s possible to collect more data in your case. If feasible, keep collecting data for the 'failure' and 'relapse' class.

2. Resampling techniques:

  • Undersampling "cured" and oversampling other classes to balance class distribution. (Link1, Link2).
  • Or use hybrid approach(both oversampling the minority class and undersampling the majority classes)

3. Loss functions:

  • Check out weighted loss: higher penalties for misclassifying examples from the minority classes. Thiss will encourage the model to focus on learning their characteristics.

4. Modeling: I believe Tree based models and ensemble methods tends work better on imbalanced data.

  • Tree based: such as Random Forests and Gradient Boosting
  • Ensemble methods: combining multiple models (e.g., bagging or boosting) can improve performance and generalization.
  1. Class-specific evaluation metrics: Apart from the regular confusion matrix, evaluate your model's performance using metrics like precision, recall, F1-score, and AUC-ROC for each class, not just overall accuracy

Do experiments. Find the best approach for your specific data. That's the way to learn!

  • $\begingroup$ Thank you so much! $\endgroup$
    – Jeff
    Commented Jun 26 at 18:22
  • $\begingroup$ It would be great if you could accept the answer if it has answered your question. $\endgroup$ Commented Jun 26 at 20:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .