# In a Recurrent Neural Network, what are the inputs to a node in a mutli-layer RNN?

I'm trying to work through a project where I'm writing my own RNN in C++ - not using any libraries. Basically I have an Input layer - 2 hidden layers - and then an output layer. In a given layer, each node collects inputs from EVERY node in the previous layer and then adding a bias, which is normal - like a normal Feed-Forward network.

But here is my question - for the Recurrence, in addition to inputs from previous layer, does a node:

• ONLY get feedback from itself
• get feedback from every node in its own layer (this is what I'm currently doing - but it may be overkill)
• or can a node get feedback from nodes in layers AFTER it - I've seen architecture diagrams that suggest this.

I'm thinking the first one is the way to go - the second one is what I'm current trying on a number of applications and very simple models just never seem to converge. I imaging the 3rd one, would be even more difficult to train.

OR are all 3 correct - they are just different types of RNNs?

Also, I'm assuming that nodes in the OUTPUT layer do not feedback to themselves - those nodes still do activation and have an array of weights and a bias, but they ONLY collect inputs from the last hidden layer - is that correct?

Thanks Mike

• Finally, an interesting question!
– nbro
May 22 at 23:35
• By "node" you mean a single unit/neuron in the recurrent layer? May 23 at 10:56
• Correct - yes a single Neuron that collects inputs, multiplies by weights, adds a bias, then passes the result into an activation function May 23 at 13:28

A node in a multi-layer RNN $$R$$ at time $$t$$, and layer $$l$$ (i.e. $$R_{t,l}$$) gets 2 inputs:
1. $$R_{t-1, l}$$: Same layer, but previous time-step. On the first time-step, you can take any (random or zero) input.
2. $$R_{t, l-1}$$: Same time-step, but previous layer. On the first layer, you take the (user-)input at time t.