There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly sequential. That is, the computation of the next tokens depends on the result of previous tokens, thus, RNNs are losing to Attention in terms of speed.
Even though I fully agree that RNNs are sequential as stated above, I think they are still parallelizable by splitting the mini-batch into sub-batches and each of these sub-batches is processed independently by a dedicated thread. For example, a training batch of size 32 can be split into 4 sub-batches of size 8; 4 threads process 4 sub-batches independently. That way, RNNs/LSTMs are parallelizable and this is not a disadvantage compared to Attention.
Is my thought correct?