I trained a binary classifier using ML.NET's AutoML feature on a small dataset (compared to other, similar models I've trained that seem to work well)-around 500 rows with around 50 features. AutoML used cross-validation with 5 folds.
The training data is balanced to about 200 positive cases to 300 negative cases, which isn't an unreasonable representation of the real world based on domain knowledge.
The model's metrics are poor compared to other, similar models, e.g.:
- Accuracy: 0.64
- Positive Precision: 0.375
- Positive Recall: 0.09
- Negative Precision: 0.67
- Negative Recall: 0.92
- F1 Score: 0.15
When the model is run against unseen data, it predicts the negative case 99% of the time.
If the accuracy were truly as stated in the metric, a correct classification 2/3 of the time has some practical value in this application. However, the actual predictions of 99% the negative case are surely flawed.
Is the training set too small to expect reasonable results? Is there anything I can do to improve the model?