1
$\begingroup$

I have an NLP model for answer-extraction. So basically I have a paragraph and a question as input and my model extracts the span of the paragraph that corresponds to the answer to the question.

I need to know how to compute F1 score for such models. It is the standard metric (along with Exact Match) used in the litterature to evaluate question-answering systems.

Thanks for your help.

$\endgroup$
0
$\begingroup$

It really depends on what you are looking for your model to do. For example do false negative or false positive really cost your research (or your business)? Also it's very important to consider your label (class) distribution.

If you just want to acheive the highest accuracy, and you don't have any issue with your class distribution (that I beleive you probably don't in your case) then accuracy works pretty good.

F1 score might be a better option to use if you need to seek a balance between precision and recall and there is an uneven class distribution.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your answer. In my case I need to compute F1 to compare my model to another one. So I need a clear definition of F1 in case of question/answering tasks like SQuAD dataset. In other words, I need to know how is F1 score calculated for a model trained on SQuAD dataset. $\endgroup$ – HLeb Aug 2 at 11:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.