How to change the backward pass for an LSTM layer that outputs to another LSTM layer?

Question

I am currently trying to understand the mathematics in Ger's paper Long Short-Term Memory in Recurrent Neural Networks. I have found the document clear and readable so far.

On pg. 21 of the pdf (pg. 13 of the paper), he derives the backward pass equations for output gates. He writes

$$\frac{\partial y^k(t)}{\partial y_{out_{j}}} e_k(t) = h(s_{c_{j}^{v}}(t)) w_{k c_{j}^{v}} \delta_{k}(t)$$.

If we replaced $\delta_{k}(t)$, the expression becomes

$$\frac{\partial y^k(t)}{\partial y_{out_{j}}} e_k(t) = h(s_{c_{j}^{v}}(t)) w_{k c_{j}^{v}} f'(net_k(t)) e_k(t)$$.

He states that the result of the partial derivative $\frac{\partial y^k(t)}{\partial y_{out_{j}}}$ comes from differentiating the forward pass equations for the output units.

From that and from the inclusion of $e_k(t)$, the paper implies that there is only one hidden LSTM layer. If there are multiple hidden LSTM layers, it wouldn't make sense.

Because if $k$ is the index of LSTM cells that the current cell is outputting to, then $e_k(t)$ would not exist since the cell output isn't compared with the target output of the network. And if $k$ is the index of output neurons, then $w_{k c_{j}^{v}}$ would not exist since the memory cells are not directly connected to output neurons. And $k$ cannot mean different things since both components are placed under a sum over $k$. Therefore, it only makes sense if the paper assumes a single LSTM layer.

So, how would one modify the backward pass derivation steps for an LSTM layer that outputs to another LSTM layer?

Answer 1

My understanding is that you have posted the last equation on p. 19 in the dissertation (please correct me if I'm wrong).

The derivation is indeed for a single LSTM layer as e_k(t) is the error at the output (the assumed network topology is also mentioned just below Eq. (2.8) in the dissertation):

Finally, assuming a layered network topology with a standard input layer, a hidden layer consisting of memory blocks, and a standard output layer[...]

Let's assume now that there are two stacked LSTM layers. In this case, the derivation in the dissertation applies to the last hidden LSTM layer (the one below the output layer), where Eqs. (3.10) -- (3.12) give you the partial derivatives for the weights at each gate for a cell in that layer. To derive the deltas for the hidden LSTM layer below, you have to compute the partial derivatives with respect to the portions of net_cv^j(t), net_inj(t) and net_fj(t) terms corresponding to the outputs of the preceding hidden LSTM layer, and then use those in the same way you used e_k(t) for the current LSTM layer. Underneath all that unappealing notation is just the usual multi-layer backprop rule (well, with the truncated RTRL twist). If you stack more LSTM layers, just keep propagating the errors further down through the respective gates until you reach the input layer.

For a slightly more intuitive explanation, if you look at Fig. 2.1 in the dissertation, you can assume that in a multi-layered LSTM the IN in fact includes the OUT of the preceding LSTM layer.

Edit

There is a nice diagram of the flow of partial derivatives here (also see subsequent slides).

In this example, all x^t terms represent the external input to that layer at time step t. In a multi-layer LSTM, this includes input from the LSTM layer below the current one. To propagate the error to the layer below, take the derivatives with respect to all gate weights corresponding to x^t and apply the chain rule.

The reason why the derivation for the output gate weights is different is that the gating is applied after the cell state is updated, whereas the cell, input and forget gates are applied before that. This matters when computing the gradients.