I need to report accuracies of my neural model in a conference paper as compared to various baselines. What are the accepted standards for reporting accuracies in a fair manner?

Neural Model: To be specific, I'm using 60% as train set, 20% as validation set and 20% as test set to report the accuracy of my neural model.

  1. Should I take an average or highest of 3 runs accuracy where in each run I randomly sample 60% as training data from the total 80% train + validation set.
  2. My neural model is computationally intensive and therefore it is not feasible to perform a k-fold cross validation. Will my accuracy results be accepted by the academic community without a k-fold cross validation? Since my data set is large, I assume using 20% of it used solely for testing should be a fair indicator of accuracy.

Bag-of-words (BOW):

  1. How do I report the accuracy for this model so as to perform a fair comparison?

  2. Should I train BOW on only 60% of data (same as which my neural model is being trained on) or should I train BOW on 80% of data (train + validation for my neural model)? Which is the accepted way? I then test BOW on the same remaining 20% of data (as in neural model) in either of the above case.

  3. The other approach is to perform k-fold cross validation but the test set will not be the same 20% as the test set on which my neural model is being evaluated. Is this approach recommended though?

Any other information on how to report accuracy results in a research paper comparing neural models (train, val, test) with linear models-BOW,SVM (train,test) is welcome. Please help.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.