Any comparison between transformer and RNN+Attention on the same dataset?

Question

I am wondering what is believed to be the reason for superiority of transformer?

I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. However, as far as I know, you can use attention also with RNN architectures as in the famous paper attention is introduced(here)).

I am wondering whether the only reason for the superiority of transformers is because they can be highly parallelized and trained on much more data?

Is there any experiment comparing transformers and RNN+attention trained on the exact same amount of data comparing the two?

Answer 1

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation method:

For example, Deep-Att + PosUnk is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the transformer with self-attention is $2.3 \cdot 10^{19}$ (FLOPs) and $1.0 \cdot 10^{20}$ (FLOPs) for the "Deep-Att + PosUnk" method (the transformer is 4 times faster) on "WMT14 English-to-French" dataset.

Please note that the BLEU is a crucial factor here (not merely training cost). Hence, you can see the BLEU‌ value of the transformer superior to the ByteNet (Neural Machine Translation in Linear Time). Although the ByteNet has not adopted the RNN, you can find the comparison of the ByteNet with other "RNN + Attention" methods in its original paper:

Hence, by transitivity property of the BLEU score, you can find that the transformer has already outperformed other "RNN‌ + Attention" methods in terms of the BLEU score (please check their performance on "WMT14" dataset).