Why is the WMT16 dataset favoured for evaluating machine translation models?

Question

The Workshop on Statistical Machine Translation has released translation challenges each year from 2004 on, which feature a dataset of sentence pairs in a variety of languages.

Even though the conference has been taking place each year, with ever growing dataset sizes, the dataset from 2016 is still being used frequently as an evaluation dataset in scientific papers when evaluating machine translation models and it has by far the most downloads on the huggingface hub.

I would be grateful if someone could shed light on the reason for which the WMT16 dataset is so strongly favoured by the machine translation community.

Answer 1

My guess is the reason for the stats on Huggingface Hub is in fact pretty random: something like that someone uses the particular dataset in a tutorial.

Usually, it is considered a good idea to use the same data as previous work a paper is compared with. WTM16 was the first one where neural MT models clearly dominated, this might be part of the reason. There were bigger datasets in later years, but until the WMT16 data will be considered really small, some people will certainly prefer more direct comparability with previous work.

The WMT14 dataset is also frequently used in papers. The original Transformer paper presented the results on WMT14 data and since that time people tend to compare their results with them.

Also that the data is bigger does not necessarily have to be a good thing. University labs do not have as big computational resources as big companies and they need to trade-off the number of experiments and the training data size. Big, but not huge dataset such as WMT16 might be a good choice.

Answer 2

Could it be also because in 2016 WMT became a conference and hence more "trustworthy" as a citation source? Until then WMT was a workshop. Maybe the older the papers, the more famous and studied they get :D

https://machinetranslate.org/wmt