0
$\begingroup$

I have a text generation model and I want to evaluate its output by comparing it to a set of gold human-annotated references.

I went through machine-translation metrics and I found that BLEU is used as the main metric usually. I didn't like using it because it's shallow as it uses ngrams comparison; the semantics of the translation is missed.

Is there any other metric to do a semantic-based evaluation?

I've thought of using a text similarity model to evaluate the output or even an NLI (Natural language inference) system. I am not sure how precise the evaluation will be because SOTA systems are not really accurate.

$\endgroup$
0
$\begingroup$

There are other possible metrics, e.g. meteor and BLEURT. They compensate some of the basic problems most researchers would like to avoid BLEU for. The downside of not using known metrics is, that your model is even harder to evaluate against other candidates. If you compare to human gold standard corpus, you should not count on BLEURT too much since it is actually intended to evaluate two systems against a gold corpus and tell you which is better.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.