# How does a transformer leverage the GPU to be trained faster than RNNs?

How does a transformer leverage the GPU to be trained faster than RNNs?

I understand the parameter space of the transformer might be significantly larger than that of the RNN. But why does the transformer structure can leverage multiple GPUs, and why does that accelerate its training?

• A recurrent network depends on the previous time steps hidden state. Therefore you can’t calculate the $t$th hidden state without first calculating the $t-1$th. This is not the case with transformers, there is no such recursive dependency — the recursive nature is capture in other ways (eg through self attention and explicitly positional encoding) Commented Nov 30, 2021 at 23:06
• @DavidIreland That seems like a good starting point for a formal answer.
– nbro
Commented Dec 1, 2021 at 12:22

A recurrent neural network (RNN) depends on the previous hidden state from the previous time step. That is, an RNN is a function of both the data for the sequence at time $$t$$ and the hidden state from time $$t-1$$. This means that we cannot compute the $$t$$th hidden state without calculating the $$t-1$$th state, and the $$t-1$$th state without the $$t-2$$th state, and so on.

In contrast to this, a transformer is able to fully parallelise the processing of the sequence because it does not have this recursive relationship, i.e. a transformer is not a recursive function -- the recursive nature of the sequence is processed in other ways, such as through positional encoding. We can see this by the way self attention works.

If we first consider the general attention mechanism framework, then we have a query $$q$$ and a set of paired key-value tuples $$\textbf{k}_1, ..., \textbf{k}_n$$ and $$\textbf{v}_1, ..., \textbf{v}_n$$. In general, for each key, we will apply some attention function $$\beta$$ (such as a neural network) to obtain attention scores, $$a_i = \beta(\textbf{q}, \textbf{k}_i)$$. We then define an attention vector $$\textbf{a}$$ where the $$i$$th element is the $$i$$th attention score, and we take a softmax of this vector to obtain attention weights $$\alpha_i$$ where $$\alpha_i$$ is the $$i$$th element of $$\mbox{softmax}(\textbf{a})$$. The output of the attention mechanism for query $$\textbf{q}$$ is then the weighted sum $$\sum_{i=1}^n \alpha_i \textbf{v}_i$$.

Now that we have the necessary background for an attention mechanism, we can look at self attention which is the backbone of Transformer. If we have a sequence denoted by $$\{\textbf{x}_1, ..., \textbf{x}_n\}$$, then we can define a set of queries, keys and values to be these $$\textbf{x}_i$$ values. Note that previously we only had a single query, but here we will have multiple queries which is really how Transformer is able to parallelise the processing of the sequence. If we define $$\textbf{Q}, \textbf{K}, \textbf{V}$$ to be the matrices of the queries, keys and values (e.g. the $$i$$th row of $$\textbf{Q}$$ corresponds to the $$i$$th query, and similarly for the others). Self attention is as simple as performing attention over these query, key and values -- the name self comes from the fact that the queries, keys and values are all the same and represent the $$i$$th element of the sequence. Now, we can write the above attention mechanism as $$a_{i, j} = \beta(Q, K)$$ where we now have a matrix of attention scores (because we have $$i$$ queries and $$i$$ keys the matrix will be square), and we can take softmax row-wise to get the attention weights (again, this will be an $$i\times i$$ matrix). If we call the matrix of attention weights $$\textbf{A}$$ then the output of a self attention layer will be given by $$\textbf{A} \textbf{V}$$. As you can see, there is no recursive nature here and this is all parallelisable, e.g. it can be broken up and put onto multiple GPU's at the same time -- this would not be possible with an RNN as you would have to wait for the output of the previous layer.

• please someone let me know if I have messed up the orders of my matrix multiplications. Commented Dec 2, 2021 at 21:11
• Are you saying that RNN/LSTM can only access to information of the previous time step (previous token), while Transformer can access to information of every other tokens in the sequence ? Thanks
– user77925
Commented Mar 12 at 0:02
• @abcd kind of, yes. The RNN has access to information from previous time steps through the hidden state, but if you look at how this is updated (in e.g. LSTMs which use a gating mechanism) then it starts to lose more information about time $t$ at time $t+k$ as $k$ increases. This isn't wholly true in transformers, as they attend to every previous token in the context window. Commented Mar 12 at 12:24
• I'm less sure on them but I think the same problem would happen. For really long sequences, the hidden state would start to lose information (I think). Commented Mar 12 at 16:54
• @abcd kind of hard to answer in a comment, probably it would require a writing out of the equations used in attention (perhaps you could ask it in a question and either myself or someone else will answer it), but basically if you look at how self-attention works, you can see that they attend to every token in the sequence without any loss of information (e.g. there is no attempt to collect all prior information in a single dynamic state) Commented Mar 13 at 20:56

The issue with Recurrent models is that they don't parallelization during training. Sequential models performs better with more memory but faces problem in learning long-term memory dependencies.

On the other hand Transformers take into account of self attention which boosts the speed of how fast the model can translate from one sequence to another and establishes dependencies b/w input and output and focus on relevant parts of the input sequence, which in turn eliminates recurrence and convolution unlike RNNs where sequential computation inhibits parallelization.

• This answer does not provide much insight into how transformers are "more parallelizable" than RNNs. You say "unlike RNNs where sequential computation inhibits parallelization", but that's not very useful because you don't explain why transformers avoid the issue and why "sequential computation" is really the problem. You talk about many things that are irrelevant to answer the question and only give a few words to answer the question, which are unclear.
– nbro
Commented Sep 17, 2020 at 14:01