Is this a correct visual representation of a recurrent neural network (RNN)?

Question

This is a picture of a recurrent neural network (RNN) found on a udemy course (Deep Learning A-Z). The axis at the bottom is "time".

In a time series problem, each yellow row from left to right would represent a sequence of a feature. In this picture, then, there are 6 sequences from 6 different features that are being fed to the network.

I am wondering if the arrows in this picture are completely accurate in an RNN. Shouldn't every yellow node also connect to every other blue node along its depth dimension? By depth dimension here I mean the third dimensional axis of the input tensor.

For example, the yellow node at the bottom left of this picture, which is closest to the viewer, should have an arrow pointing to all the blue nodes in the array of blue nodes that is at the very left, and not just to the blue node directly above it.

Answer 1

I will answer this question, leaving it open to challenge by anyone more knowledgeable.

The equations to update each layer of an RNN are

$y_t = \sigma(W_x x_t + b_x )$

and

$h_t = \sigma(W_hx_t + b_h)$

Where $h_t$ is the hidden layer (in blue in the picture), and $y_t$ is the output layer (red in picture). This equations says that every single component in the hidden layer vector, i.e. every unit in the hidden layer, is a function of the linear combination of the $x_t$ vector, which is the first yellow row along the depth axis. In other words, all the first yellow input nodes on the bottom left in the picture.

Thus, technically this picture is not correct, all the yellow nodes should have points to all the blue nodes. Also, by similar reasoning, all the blue nodes in the subsequent steps of the hidden layer should be connected to all the blue nodes of the previous layer.

Of course, that would make for a much uglier/harder picture to make, so I don't blame the authors, although this has given me a few hours of confused research, which I guess still meets their educational goal.