Suppose we have a data set with $4,000$ labeled examples. The outcome variable is trinary (three possible categorical values). Suppose the accuracy of a given model is "bad" (e.g. less than $50 \%$).

Question. Should you try different traditional machine learning models (e.g. multinomial logistic regression, random forests, xgboost, etc.), get more data, or try various deep learning models like convolutional neural networks or reccurent neural networks?

If the purpose is to minimize time and effort in collecting training data, would deep learning models be a viable option over traditional machine learning models in this case?

  • $\begingroup$ This post may need more details to be answered properly. What model were you using at the time? What were its hyper-parameters? What loss was you using? $\endgroup$ – nbro Nov 5 '20 at 10:47

To know if your model needs more training data, try to plot out "learning curves", that are based on increasing size of the training set.

Basically, you calculate training and validation accuracy metrics for 1, 2, 3, 4, 5, ..., m training samples. Size of validation set may be constant over time. If the accuracy is still rising when your data set is fully used, then you need more training data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.