3
$\begingroup$

I'm trying to work through a project where I'm writing my own RNN in C++ - not using any libraries. Basically I have an Input layer - 2 hidden layers - and then an output layer. In a given layer, each node collects inputs from EVERY node in the previous layer and then adding a bias, which is normal - like a normal Feed-Forward network.

But here is my question - for the Recurrence, in addition to inputs from previous layer, does a node:

  • ONLY get feedback from itself
  • get feedback from every node in its own layer (this is what I'm currently doing - but it may be overkill)
  • or can a node get feedback from nodes in layers AFTER it - I've seen architecture diagrams that suggest this.

I'm thinking the first one is the way to go - the second one is what I'm current trying on a number of applications and very simple models just never seem to converge. I imaging the 3rd one, would be even more difficult to train.

OR are all 3 correct - they are just different types of RNNs?

Also, I'm assuming that nodes in the OUTPUT layer do not feedback to themselves - those nodes still do activation and have an array of weights and a bias, but they ONLY collect inputs from the last hidden layer - is that correct?

Thanks Mike

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Finally, an interesting question! $\endgroup$
    – nbro
    Commented May 22, 2023 at 23:35
  • $\begingroup$ By "node" you mean a single unit/neuron in the recurrent layer? $\endgroup$
    – Luca Anzalone
    Commented May 23, 2023 at 10:56
  • $\begingroup$ Correct - yes a single Neuron that collects inputs, multiplies by weights, adds a bias, then passes the result into an activation function $\endgroup$
    – Mike Arsenault
    Commented May 23, 2023 at 13:28

1 Answer 1

Reset to default
2
$\begingroup$

If I understand you correctly, then option one is the way multi-layer RNNs are usually implemented.

A node in a multi-layer RNN $R$ at time $t$, and layer $l$ (i.e. $R_{t,l}$) gets 2 inputs:

  1. $R_{t-1, l}$: Same layer, but previous time-step. On the first time-step, you can take any (random or zero) input.
  2. $R_{t, l-1}$: Same time-step, but previous layer. On the first layer, you take the (user-)input at time t.

This is the standard implementation. However, I don't doubt that you can do many different things, which might or might not work in certain situations. Think of bidirectional RNNs etc.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thank you!! Yes - I've often thought that there is no INCORRECT way of doing it. I've tried something like this in the past - the SINGLE hidden layer - is just a mass collection of neurons - each one is connected to the input layer and provides outputs to the output layer - and then each neuron is connected to EVERY other neuron in this mass collection - picture a place of spaghetti :-). I think this didn't converge at all (or didn't seem to) and then I read that there is some benefit to formally pass values (or outputs) from one layer to the next, sequentially $\endgroup$
    – Mike Arsenault
    Commented May 23, 2023 at 13:41
  • $\begingroup$ Glad the answer helped you! If it completely solves your question, please check it as 'the correct answer' by assigning the checkmark to it. This helps future visitors of the site find what they are looking for :) $\endgroup$
    – Robin van Hoorn
    Commented May 23, 2023 at 14:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .