You are right. This is a broad question, but I'll try to provide some general direction. Let's go through the whole pipeline from data collection to model selection, training, and evaluation. These are just basic things to consider. Choose and experiment based on your dataset. With limited data, it will be tough to generalize, but here is checklist:
1. Data collection: I'm not sure if it’s possible to collect more data in your case. If feasible, keep collecting data for the 'failure' and 'relapse' class.
2. Resampling techniques:
- Undersampling "cured" and oversampling other classes to balance class distribution. (Link1, Link2).
- Or use hybrid approach(both oversampling the minority class and undersampling the majority classes)
3. Loss functions:
- Check out weighted loss: higher penalties for misclassifying examples from the minority classes. Thiss will encourage the model to focus on learning their characteristics.
4. Modeling: I believe Tree based models and ensemble methods tends work better on imbalanced data.
- Tree based: such as Random Forests and Gradient Boosting
- Ensemble methods: combining multiple models (e.g., bagging or boosting) can improve performance and generalization.
- Class-specific evaluation metrics:
Apart from the regular confusion matrix, evaluate your model's performance using metrics like precision, recall, F1-score, and AUC-ROC for each class, not just overall accuracy
Do experiments. Find the best approach for your specific data. That's the way to learn!
