# How to graphically represent a RNN architecture implemented in Keras?

I'm trying to create a simple blogpost on RNNs, that should give a better insight into how they work in Keras. Let's say:

model = keras.models.Sequential()
model.add(keras.layers.SimpleRNN(5, return_sequences=True, input_shape=[None, 1]))


I came up with the following visualization (this is only a sketch), which I'm quite unsure about:

The RNN architecture is comprised of 3 layers represented in the picture.

Question: is this correct? Is the input "flowing" thought each layer neuron to neuron or only though the layers, like in the picture below. Is there anything else that is not correct - any other visualizations to look into?

Update: my assumptions are based on my understanding from what I saw in Geron's book. The recurrent neurons are connected, see: https://pasteboard.co/JDXTFVw.png ... he then proceeds to talk about connections between different layers, see: https://pasteboard.co/JDXTXcz.png - did I misunderstand him or is it just a peculiarity in keras framework?

• As you can see in fig 15-2, neurons in a layer don't communicate with each other, it is unrolled version in the 2nd part of 15-2, ie. it is the same layer across each timestep, there is no next layer in that diagram. In fig 15-1, a single neuron is shown (unrolled on right). What's happening for a single neuron in 15-1 is happening to every neuron in 15-2, there is no concept of multiple layers introduced in the images you referenced, they are different timesteps for the same neuron/layer. Dec 8 '20 at 11:49
• Yea, I was getting the 2nd part of the argument you presented, now I also understand the first one - the confusion creeped in because the example was using a single neuron RNN as a simplified self-contained network and then added several cells in a single layer RNN. As you said - both pictures show single layer RNN unrolled through time, the difference being the first one has a single neuron and the other has multiple neurons in that layer. Thanks Dec 8 '20 at 12:45