4
$\begingroup$

In the lecture, there was a statement:

"Recurrent neural networks with multiple hidden layers are just a special case that has some of the hidden to hidden connections missing."

I understand recurrent means that can have connections to the previous layer and the same layer as well. Is there a visualization available to easily understand the above statement?

$\endgroup$
4
$\begingroup$

I assume the statement was made for Elman recurrent neural networks, because as far as I know, that is the only type of neural networks for which that statement is valid.

Let's say we have an Elman recurrent neural network with one input neuron, one output neuron and one hidden layer with two neurons. Elman recurrent neural network In total there are 10 connections. As the image shows, neuron A receives the combined previous output of both neuron A and B as input. The same goes for neuron B.

This is not the case when we split the neurons up into multiple layers; the context neuron(s) are only used by neurons that are in the same layer. Let say we now use multiple hidden layers and keep the amount of neurons the same. In total there are 7 connections now (image below). That is 3 less than in the first example, which has only one hidden layer. So which connections do we miss? That is shown in the bottom image. (I had to paste these two images together in one image because my reputation only allows me to post 2 links) The difference between with and without multiple hidden layers

Please note the cross; the connection between neuron A and B is not there in the first image, because it would be some kind of random recurrent connection.

The first and the last image are exactly the same. I think that if you compare the first and the last image that you agree that the statement is true.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.