2
$\begingroup$

enter image description here

This is a picture of a recurrent neural network (RNN) found on a udemy course (Deep Learning A-Z). The axis at the bottom is "time".

In a time series problem, each yellow row from left to right would represent a sequence of a feature. In this picture, then, there are 6 sequences from 6 different features that are being fed to the network.

I am wondering if the arrows in this picture are completely accurate in an RNN. Shouldn't every yellow node also connect to every other blue node along its depth dimension? By depth dimension here I mean the third dimensional axis of the input tensor.

For example, the yellow node at the bottom left of this picture, which is closest to the viewer, should have an arrow pointing to all the blue nodes in the array of blue nodes that is at the very left, and not just to the blue node directly above it.

$\endgroup$
  • $\begingroup$ On a different note, udemy is a pretty sketchy educational platform (with many plagiarized courses) and not very credible instructors. It's better if you take courses from experts like those in stanford, MIT, etc which are freely available mostly. $\endgroup$ – DuttaA Apr 7 at 5:00
  • $\begingroup$ @DuttaA Thanks for the heads up, those courses definitely don't seem to be very rigorous. $\endgroup$ – Mike Apr 7 at 5:19
  • $\begingroup$ which ones...The udemy ones? $\endgroup$ – DuttaA Apr 7 at 5:31
  • $\begingroup$ @DuttaA Yeah, I was referring to the udemy ones. $\endgroup$ – Mike Apr 10 at 19:50
0
$\begingroup$

I will answer this question, leaving it open to challenge by anyone more knowledgeable.

The equations to update each layer of an RNN are

$y_t = \sigma(W_x x_t + b_x )$

and

$h_t = \sigma(W_hx_t + b_h)$

Where $h_t$ is the hidden layer (in blue in the picture), and $y_t$ is the output layer (red in picture). This equations says that every single component in the hidden layer vector, i.e. every unit in the hidden layer, is a function of the linear combination of the $x_t$ vector, which is the first yellow row along the depth axis. In other words, all the first yellow input nodes on the bottom left in the picture.

Thus, technically this picture is not correct, all the yellow nodes should have points to all the blue nodes. Also, by similar reasoning, all the blue nodes in the subsequent steps of the hidden layer should be connected to all the blue nodes of the previous layer.

Of course, that would make for a much uglier/harder picture to make, so I don't blame the authors, although this has given me a few hours of confused research, which I guess still meets their educational goal.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.