In the paper https://arxiv.org/pdf/1211.5063 the authors provide an alternative equation for the more widely known equation to calculate the hidden state at timestep $t$ $$ x_t = σ(W_{rec}x_{t−1} + W_{in}u_t + b) $$

which is $$ x_t = W_{rec}σ(x_{t−1}) + W_{in}u_t + b $$

They claim

This formulation is equivalent to the more widely known equation xt = σ(Wrecxt−1 + Winut + b), and it was chosen for convenience.

I don't see how those are equivalent. When applying the non-linearity to the whole expression you are certainly calculating something else than just applying it to the last hidden state.

Can someone explain why those two are equivalent in this scenario?

  • $\begingroup$ Could you please also format the formulas in the quote with mathjax? Thanks. $\endgroup$
    – nbro
    Commented Jun 15 at 2:09


You must log in to answer this question.

Browse other questions tagged .