# Any comparison between transformer and RNN+Attention on the same dataset?

I am wondering what is believed to be the reason for superiority of transformer?

I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. However, as far as I know, you can use attention also with RNN architectures as in the famous paper attention is introduced(here)).

I am wondering whether the only reason for the superiority of transformers is because they can be highly parallelized and trained on much more data?

Is there any experiment comparing transformers and RNN+attention trained on the exact same amount of data comparing the two?

• Hi. Welcome to AI SE! Please, take a look at ai.stackexchange.com/help/on-topic to know more about our site and its scope, in case you haven't done it yet. Regarding your question, have you already looked at the original paper that introduced the transformer, i.e. "Attention is all you need"? They compare the transformer to other models that use RNNs (with attention, if I recall correctly). – nbro Oct 5 '20 at 8:00
• Thanks @nbro! I did not know about the exact design of the other models. The provided answer answered my question completely. Thanks! – milad aghajohari Oct 12 '20 at 23:09

For example, Deep-Att + PosUnk is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the transformer with self-attention is $$2.3 \cdot 10^{19}$$ (FLOPs) and $$1.0 \cdot 10^{20}$$ (FLOPs) for the "Deep-Att + PosUnk" method (the transformer is 4 times faster) on "WMT14 English-to-French" dataset.