If the accuracy of my current model is low ($50 \%$) and we want to minimize time in collecting more data, should we try other models?

Question

Suppose we have a data set with $4,000$ labeled examples. The outcome variable is trinary (three possible categorical values). Suppose the accuracy of a given model is "bad" (e.g. less than $50 \%$).

Question. Should you try different traditional machine learning models (e.g. multinomial logistic regression, random forests, XGBoost, etc.), get more data, or try various deep learning models like convolutional neural networks or recurrent neural networks?

If the purpose is to minimize time and effort in collecting training data, would deep learning models be a viable option over traditional machine learning models in this case?

Answer 1

To know if your model needs more training data, try to plot out "learning curves", that are based on increasing size of the training set.

Basically, you calculate training and validation accuracy metrics for 1, 2, 3, 4, 5, ..., m training samples. Size of validation set may be constant over time. If the accuracy is still rising when your data set is fully used, then you need more training data.

Answer 2

Accuracy can sometimes be a very coarse metric. When it is applied to three class problems, people often take the class label with maximum predicted probability and predict that. The probabilities of the individual labels are ignored. I'd recommend that as well as accuracy you calculate sensitivity and specificity for each class and the area under the ROC curve. For both of these, you can take a 1 vs rest (i.e. class 1 vs classes 2 & 3, class 2 vs classes 1 & 3 etc) approach to calculating the metrics. Even if your accuracy is under 50%, the model may still be predicting at least one class well, so I recommend doing more analysis before making a decision.

I also recommend comparing your model (if it is deep learning-based) to a traditional type of model, as deep learning usually works best with big data and with such a small dataset as you have, there may be no benefit to using deep learning (unless you are leveraging transfer learning).

Answer 3

If data collection is expensive, it is better to first try to improve your model.

You say your accuracy is bad, but have you using tried better performance metrics? A confusion matrix could help. Another potential problem is that your data may be imbalanced. What if your model is performing badly because, for example, there aren't enough samples from class 2. Now you know that to improve your model you need mode class 2 samples. This can be acquired by collecting more data, or by other class imbalance methods e.g. SMOTE.

You can also check whether the performance is bad only on the test set. Is the model overfitting? Or does it perform badly on the training set too? Underfitting. 4000 samples should be enough for a neural network, but not a very large deep learning model, maybe a smaller one. Testing other machine learning models on your data is definitely a good idea. Not only you can provide your model with a benchmark, but you also may find a better model.

If you think your data is good enough and your model is adequate, you can try hyperparameter optimization. This can be quire useful for getting the most out of neural networks, but it is trial and error, and it can take some time to find the best hyperparameters.