Need Help With LSTM Neural Networks

Question

I have been researching LSTM neural networks. I have seen this diagram a lot and I have few questions about it. Firstly, is this diagram used for most LSTM neural networks?

Secondly, if it is, wouldn't only having single layers reduce it's usefulness?

Answer 1

Just to be 100% sure - the diagram you refer to is a diagram of an LSTM CELL, not NETWORK. The operands you see on the diagram are operations within a cell, not separate "neurons". I think it is quite obvious, however reading your questions I just wanted to be 100% sure we are on the same page.

Now, about layers. RNN networks (LSTM in particular) are just like any other ANN structure. Theoretically, a 1-hidden layer network can do any computation of a "deeper" network. ANN is a universal approximate of math functions. Still, multi-layer ANNs typically work better on more complex problems. Multi-layer typically needs less total connections, learns better and is less resource-demanding.

In particular, multi-layer LSTMs are believed to be better at determining complex in-time patterns. I think there is no rigorous proof for this, however. Also, in practical applications - I did not see much improvement in network capabilities by adding additional LSTM layers. Adding more dense layers before/ after LSTM seemed to have a much better effect.

Answer 2

(1) Yes this is the diagram for a classical LSTM unit. Of cause there are some variants and those diagrams would look slightly different.

(2) It is very common for researchers to use more than one layers of LSTM and achieves better performance than a single layer one. A common way to "stack" LSTMs is to use the previous layer's output ($h_t$ in your diagram) as the input to the next layer ($x_t$). However, I have seldom seen any successful application of 5+ layers of LSTMs, while for CNNs it is common to use tens or even hundreds of layers.