What are the differences between BLEU and METEOR?

Question

I am trying to understand the concept of evaluating the machine translation evaluation scores.

I understand how what BLEU score is trying to achieve. It looks into different n-grams like BLEU-1,BLEU-2, BLEU-3, BLEU-4 and try to match with the human written translation.

However, I can't really understand what METEOR score is for evaluating MT quality. I am trying understand the rationale intuitively. I am already looking into different blog post but can't really figure out.

How are these two evaluation metrics different and how are they relevant?

Answer 1

Both BLEU and METEOR are meant to evaluate the overall translation quality. METEOR shows a slightly better correlation with human judgment than BLEU, however, it relies on n-gram alignment between the translation hypothesis and reference that needs language-specific paraphrase tables. The quality of the table heavily influences the evaluation quality. I think BLEU was preferred because of its simplicity and language independence. Nowadays, there are many evaluation metrics that correlate much better with human judgment (such as BLEURT, BertScore, or COMET) and they start to be preferred in the MT community (cf. papers: Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine ranslation Evaluation Metrics, To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation).

BLEU score is an average of n-gram precisions, weighted by the so-called brevity penalty that penalizes short high-precision, but low-recall hypotheses.

METEOR computes both precision and recall. Here, the precision corresponds to a proportion of words that are in the hypothesis and are correct. The recall is a ratio of how many words from the translation hypothesis appeared in the hypothesis. To do so, we need to somehow decide which words are correct. This is done by n-gram alignment between the hypothesis and the reference. If there is no exact match between the two, a table of paraphrases is used. To penalize sentences that contain the correct words, but in the wrong order, there is a reordering penalty term that penalizes such sentences.

Both metrics measure the same aspects of the translation, but slightly differently:

1. Precision

   • BLEU: directly via n-gram precision
   • METEOR: directly in the alignment graph

2. Recall

   • BLEU: indirectly via including the brevity penalty
   • METEOR: directly in the alignment graph

3. Fluency

   • BLEU: directly by considering longer n-grams
   • METEOR: indirectly from the properties of the alignment graph

Answer 2

BLEU is a widely-used metric for evaluating the quality of a machine translation output by measuring its correlation to reference translations. It is based on n-gram precision.

https://machinetranslate.org/bleu

METEOR, on the other hand, is a more advanced metric that also compares a machine translation output to reference translations, but it also takes into account additional information such as synonyms, word forms, and sentence structure.

https://machinetranslate.org/meteor