There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly sequential. That is, the computation of the next tokens depends on the result of previous tokens, thus, RNNs are losing to Attention in terms of speed.

Even though I fully agree that RNNs are sequential as stated above, I think they are still parallelizable by splitting the mini-batch into sub-batches and each of these sub-batches is processed independently by a dedicated thread. For example, a training batch of size 32 can be split into 4 sub-batches of size 8; 4 threads process 4 sub-batches independently. That way, RNNs/LSTMs are parallelizable and this is not a disadvantage compared to Attention.

Is my thought correct?


1 Answer 1


You are talking about model parallelism. But, that's not the reason RNNs/LSTMs are not in vogue.

Imagine your ability to read the first line of a page and going on reading and still making connections to the first line until the end of the page.

Can RNNs/LSTMs do that? No. Can Attention (i.e. Transformers) do it? Yes.

The reason is simple Attention is effectively an affinity matrix between each and every input that goes into a network. So, it is able to do that. We have a huge memory overload but hey, we want the performance.

In case of RNNs/LSTMs, the cells have to do this heavy-lifting, there is only a set amount of information that can be contained in them. That's why you have to forget gate to control information retained.

Nevertheless, your thought is correct but that's not the reason for Attention to be in vogue. But, your thought has negative ramifications when we see how to implement it. Also, nevertheless the computation will be still sequential since you can't process input (n + 1) without input n. Local parallelization is possible but not global.

  • $\begingroup$ Thank you, Abhishek, for your answer! Regarding the implementation of parallelization, could you specify in a bit more detail why it would be more complex to do so in batch-dimension (in the case of RNNs/LSTMs) compared to how parallelization is done in the case of Attention? $\endgroup$
    – zock
    Commented Apr 7, 2021 at 7:28
  • $\begingroup$ leimao.github.io/blog/Data-Parallelism-vs-Model-Paralelism $\endgroup$ Commented Apr 7, 2021 at 7:56
  • 1
    $\begingroup$ Hi Abhishek, thanks for the link but I cannot fully understand your point. I don't know if Model parallelism is often used for Attention or not, but data parallelism seems simpler than model parallelism to me. On huggingface pre-trained models, they also make use of data parallelism and support sending sub-batches to each GPU/CPU device. $\endgroup$
    – zock
    Commented Apr 7, 2021 at 8:35
  • $\begingroup$ That's exactly the point. What you were talking about is model parallelism which is hard because it is difficult to consolidate back the weights calculated on different workers. $\endgroup$ Commented Apr 7, 2021 at 8:41
  • 1
    $\begingroup$ hmm, please correct me if I am wrong, I think I was talking about using RNNs/LSTMs with data parallelism, not model parallelism, isn't it? $\endgroup$
    – zock
    Commented Apr 7, 2021 at 8:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .