How do I statistically evaluate a ML model?

Question

I have a model that predicts sentiment of tweets. Are there any standard procedures to evaluate such a model in terms of its output?

I could sample the output, work out which are correctly predicted by hand, and count true and false positives and negatives but is there a better way?

I know about test and training sets and metrics like AUROC and AUPRC which evaluate the model based on known data, but I am interested in the step afterwards when we don't know the actual values we are predicting. I could use the same metrics, I suppose, but everything would need to be done by hand.

Answer 1

There are a lot of ways to evaluate the performance of an ML model. You mentioned AUROC and AUPRC. Generally you start with the confusion matrix and derive metrics such as sensitivity, accuracy, recall, precision, etc. You can see a good outline of them here.

It seems what you are asking is a shortcut to determining how good your sentiment classification model is but there aren't any without labeled test data. You either do this by hand or you find a test set in the world, preferably something that is well know and documented and also fits your objectives. I recommend you read Neil Slater's answer at https://datascience.stackexchange.com/questions/12226/how-do-i-assess-which-sentiment-classifier-is-best-for-my-project/12228. He gives some good advice on the subjectivity of sentiment analysis classification and points out a labeled data set of Tweets which you might be able to use to test your classifier.

I also found this Kaggle competition which has a test set that might be of help to you: Angry Tweets