# Is the working of RNNs, LSTM and GRU sequential or parallel?

You take any blog or any example and all they tell you about is the given picture below.

It has 4 different matrices and 3 of whose weights are shared. So, I'm wondering how is this achieved in practice?

I think the first word "hello" goes in as a one-hot encoded form and changes the Hidden matrix. And then after it, "world" goes and gets multiplied and then changes the matrix again and so on. What people make it look like is that all of the words going are in Parallel. It can't be the case because the Hidden matrix is dependent on the previous word and without changing the matric, you can not pass the current word. Please correct if my idea is wrong but I think the execution is in sequential order.

Yes, you are correct and it was one of original motivations, which inspired the invention of the Attention mechanism in seq2seq problems https://arxiv.org/pdf/1706.03762.pdf.

There is a quote from this paper:

Recurrent models typically factor computation along the symbol positions of the input and output sequences. Aligning the positions to steps in computation time, they generate a sequence of hidden states $$h_t$$, as a function of the previous hidden state $$h_{t−1}$$ and the input for position $$t$$. This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples.

On the other hand, Transformer architectures have a loot loom for parallelization, because they take the whole sequence at once, and multiple heads can be executed in parallel.

• Thanks man! And one more thing, apart from the H (Hiddne State MAtrix) here, everything else is shared. WhX, WhY, F are all shared. Right? Jan 25 at 3:41
• And also, All of these 4 matrices have the same dimensionality which we define by (Number of Neurons) right? Jan 25 at 4:37
• WhX, WhY, F are shared. In fact you have only one cell - only the inputs x_t and h_t change. However, I would not call F a matrix - it is usually an activation function. About the dimensinality - no, in general setting. The dimension of the output y_t doesn't need to be equal to x_t or h_t. And x_t and h_t can have different dimensionality as well. This WhX and WhY are of different size. Jan 25 at 7:46
• So how can you decide the size of WhX and WhY? Also, if F is an activation function, then there are only 3 matrices not 4 right? But I thought there are 4 matrices one of which has all 0s in starting and is responsible for the state change? Right? See this link 4 matrices and 2 biases. Jan 25 at 9:51
• In your link I see only 3 matrices. There is a matrix Wxh which mutliplies the input and is added to form the hidde state, matrix Whh from hidden state to hidden state, and the matrix Why from hidden state to output y. x, h, y can be of different dimensionality. Thus, Wxh is a dim_x * dim_h matrix, Whh is a dim_h * dim_h and so on. Jan 25 at 13:14