I am writing my own recurrent neural network in Java to understand the inner workings better. While working through the math, I found that in timesteps later than 2 the gradient of weight w of neuron n depends on the gradients of all neurons at all timesteps before. A handwritten example is given, I tried to write as clearly as possible.

Could anyone verify this so I can finish my network? Am I missing a piece or is my premise wrong, as in the output of a neuron is s(Wx + Vh + b), where h is the last step of only neuron n?

The feedforward formulas for the first three timesteps My derivations of the gradients of the specific weight 1 of neuron 1, just as an example

  • $\begingroup$ are you sure that is not your homework? why are you doing it in java python makes life easier. $\endgroup$ – Steve Okay Aug 5 '17 at 8:05
  • 2
    $\begingroup$ Yes, I'm sure that this is not my homework, in fact I don't have any course related to this subject :). And firstly, Java is my preferred language, I can use other languages but I like Java the most. And speed i no concern for me. Secondly, I want to use this in another application which has to be in Java, so its just easier. Python may make dealing with arrays and numpy dealing with tensors easier, but I already got fairly good workarounds for that. $\endgroup$ – JustAGuy Aug 5 '17 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.