How does transformer leverage GPU which trains faster than RNN? I understand the parameter space of the transformer might be significantly larger than that of the RNN. But why does the transformer structure can leverage multi-gpu and accelerates its training?
The issue with Recurrent models is that they don't parallelization during training. Sequential models performs better with more memory but faces problem in learning long-term memory dependencies.
On the other hand Transformers take into account of self attention which boosts the speed of how fast the model can translate from one sequence to another and establishes dependencies b/w input and output and focus on relevant parts of the input sequence, which in turn eliminates recurrence and convolution unlike RNNs where sequential computation inhibits parallelization.