What is the output value of the network for these inputs respectively, and why? (Linear activation function is fine.)

[2, 3][-1, 2][1, 0][3, 4]

My main question is how you take the 'backwards' directed paths into account.

enter image description here


The neural Network in the image is a "Recurrent Neural Network"(RNN). Because of the connection leading backward from h10 to h01, h10 has to be a "memory node" (mn), meaning it can store its value from the previous input. The basic functionality of an RNN can be seen in this animation:


Your example:

In the beginning, the storage of the mn is initialized with a value, probably 0.
Now the first input is fed into network:

  • i0 = 2
  • i1 = 3
  • h00 = (i0 * 0.4) = 0.8
  • h01 = (i1 * -0.9) + ("the stored value of h10" * 1.2) = -2.7
    ("the stored value of h10" in the first run is 0.)
  • h10 = (h00 * 0.85) + (h01 * -0.2) = 1.22
  • out = (h10 * 0.3) + (h01 * 0.1) = 0.096

Now you can feed the next input through the network and use -2.7 as "the stored value of h10" and so on. You can also add an activation function as you would for any other NN.

  • $\begingroup$ Why is out calculated later than h10? Technically, out and h10 is on the same layer, just like h00 is to h01. What is the algorithm for the calculation order? $\endgroup$
    – andras
    Mar 31 '18 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.