3
$\begingroup$

I am wondering what is believed to be the reason for superiority of transformer?

I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. However, as far as I know, you can use attention also with RNN architectures as in the famous paper attention is introduced(here)).

I am wondering whether the only reason for the superiority of transformers is because they can be highly parallelized and trained on much more data?

Is there any experiment comparing transformers and RNN+attention trained on the exact same amount of data comparing the two?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Hi. Welcome to AI SE! Please, take a look at ai.stackexchange.com/help/on-topic to know more about our site and its scope, in case you haven't done it yet. Regarding your question, have you already looked at the original paper that introduced the transformer, i.e. "Attention is all you need"? They compare the transformer to other models that use RNNs (with attention, if I recall correctly). $\endgroup$ – nbro♦ Oct 5 '20 at 8:00
  • 1
    $\begingroup$ Thanks @nbro! I did not know about the exact design of the other models. The provided answer answered my question completely. Thanks! $\endgroup$ – milad aghajohari Oct 12 '20 at 23:09
3
$\begingroup$

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation method:

enter image description here

For example, Deep-Att + PosUnk is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the transformer with self-attention is $2.3 \cdot 10^{19}$ (FLOPs) and $1.0 \cdot 10^{20}$ (FLOPs) for the "Deep-Att + PosUnk" method (the transformer is 4 times faster) on "WMT14 English-to-French" dataset.

Please note that the BLEU is a crucial factor here (not merely training cost). Hence, you can see the BLEU‌ value of the transformer superior to the ByteNet (Neural Machine Translation in Linear Time). Although the ByteNet has not adopted the RNN, you can find the comparison of the ByteNet with other "RNN + Attention" methods in its original paper:

enter image description here

Hence, by transitivity property of the BLEU score, you can find that the transformer has already outperformed other "RNN‌ + Attention" methods in terms of the BLEU score (please check their performance on "WMT14" dataset).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks @OmG! Your answer is what I was looking for! Thank you. By the way, what if the dataset is limited? $\endgroup$ – milad aghajohari Oct 12 '20 at 23:10
  • 1
    $\begingroup$ My pleasure @miladaghajohari . Due to the translation task's high complexity, the required model's VC dimension will be increased. Hence, based on the computational learning theory, the required sample complexity to train the model will be increased as well. Therefore, a limitted dataset is unlikely to be suitable for the task. $\endgroup$ – OmG Oct 13 '20 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.