I'm trying to create a simple blogpost on RNNs, that should give a better insight into how they work in Keras. Let's say:

model = keras.models.Sequential()
model.add(keras.layers.SimpleRNN(5, return_sequences=True, input_shape=[None, 1]))
model.add(keras.layers.SimpleRNN(5, return_sequences=True))

I came up with the following visualization (this is only a sketch), which I'm quite unsure about:

enter image description here

The RNN architecture is comprised of 3 layers represented in the picture.

Question: is this correct? Is the input "flowing" thought each layer neuron to neuron or only though the layers, like in the picture below. Is there anything else that is not correct - any other visualizations to look into?

enter image description here

Update: my assumptions are based on my understanding from what I saw in Geron's book. The recurrent neurons are connected, see: https://pasteboard.co/JDXTFVw.png ... he then proceeds to talk about connections between different layers, see: https://pasteboard.co/JDXTXcz.png - did I misunderstand him or is it just a peculiarity in keras framework?

  • 1
    $\begingroup$ As you can see in fig 15-2, neurons in a layer don't communicate with each other, it is unrolled version in the 2nd part of 15-2, ie. it is the same layer across each timestep, there is no next layer in that diagram. In fig 15-1, a single neuron is shown (unrolled on right). What's happening for a single neuron in 15-1 is happening to every neuron in 15-2, there is no concept of multiple layers introduced in the images you referenced, they are different timesteps for the same neuron/layer. $\endgroup$ – SajanGohil Dec 8 '20 at 11:49
  • 1
    $\begingroup$ Yea, I was getting the 2nd part of the argument you presented, now I also understand the first one - the confusion creeped in because the example was using a single neuron RNN as a simplified self-contained network and then added several cells in a single layer RNN. As you said - both pictures show single layer RNN unrolled through time, the difference being the first one has a single neuron and the other has multiple neurons in that layer. Thanks $\endgroup$ – Mindaugas Bernatavičius Dec 8 '20 at 12:45

The first image is correct. The information will flow from left to right in each layer and from top to bottom in between layers.

  • $\begingroup$ Thanks for the comment I take it that there should be 5 nodes in the picture, not 4? Also good point on the hidden state - I think I'm missing this part as well - is the hidden state passed to the dense layer and the y value (prediction)? $\endgroup$ – Mindaugas Bernatavičius Dec 8 '20 at 10:59
  • $\begingroup$ @SajanGohil yes you are right, I haven't noticed the return sequence=True on the second layer. I will update the answer. $\endgroup$ – razvanc92 Dec 8 '20 at 11:42
  • $\begingroup$ @MindaugasBernatavičius Yes there should had been 5 nodes instead of 4. I have removed the picture since it's not mandatory anymore. Yes the hidden state is passed to the Dense layer. If you select return_sequences=True, each hidden state from the last RNN layer is passed through the dense layer. If you set return_sequences to false, then only the last hidden state from the last RNN layer is sent to the Dense layer. $\endgroup$ – razvanc92 Dec 8 '20 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.