I've just started learning LSTM, and some points in the process of calculating the gradients are getting me confused.

Say, for example, we want to compute $\frac{\partial}{\partial W_i}L$, where $L$ is the loss for one input sequence $x$ , $W_i$ is the matrix of parameters to learn in the input gate $i_t$, and $h_{t-1}$ the previous output from last time step as in the equation :

$$i_t = \sigma(W_i h_{t-1} + U_i x(t) + b_i)$$

At some point, during backpropagation, we need to do the "straightforward task", computing $\frac{\partial}{\partial W_i}i_t$ which is equal to:

$$\frac{\partial}{\partial W_i}i_t = \sigma(z_{i,t}) (1 - \sigma(z_{i,t})) h_{t-1},$$

where $$z_{i,t} = W_i h_{t-1} + U_i x(t) + b_i$$

Well, my problem is here, I do think that $h_{t-1}$ also depends on $W_i$ because $h_{t-1}$ was computed with $W_i$ (and other parameters) during the last time step. So, maybe, some more term should be added to the equation of $\frac{\partial}{\partial W_i}i_t$ above.

I hope you can understand my issue.

  • $\begingroup$ It doesn't seem to me that you provide any equation that defines $h_{t-1}$, so am I missing something here? It's been a while since I had to deal with these details, so it's likely I'm missing something. It may be a good idea to link to a reference that shows and explains in detail the equations that you're showing here. For example, where did you get this equation $\frac{\partial}{\partial W_i}i_t = \sigma(z_{i,t}) (1 - \sigma(z_{i,t})) h_{t-1},$ from? $\endgroup$
    – nbro
    Jan 8 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.