How does transformer leverage GPU which trains faster than RNN? I understand the parameter space of the transformer might be significantly larger than that of the RNN. But why does the transformer structure can leverage multi-gpu and accelerates its training?


The issue with Recurrent models is that they don't parallelization during training. Sequential models performs better with more memory but faces problem in learning long-term memory dependencies.

On the other hand Transformers take into account of self attention which boosts the speed of how fast the model can translate from one sequence to another and establishes dependencies b/w input and output and focus on relevant parts of the input sequence, which in turn eliminates recurrence and convolution unlike RNNs where sequential computation inhibits parallelization.

  • $\begingroup$ This answer does not provide much insight into how transformers are "more parallelizable" than RNNs. You say "unlike RNNs where sequential computation inhibits parallelization", but that's not very useful because you don't explain why transformers avoid the issue and why "sequential computation" is really the problem. You talk about many things that are irrelevant to answer the question and only give a few words to answer the question, which are unclear. $\endgroup$ – nbro Sep 17 '20 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.