Just to be 100% sure - the diagram you refer to is a diagram of an LSTM CELL, not NETWORK. The operands you see on the diagram are operations within a cell, not separate "neurons". I think it is quite obvious, however reading your questions I just wanted to be 100% sure we are on the same page.
Now, about layers. RNN networks (LSTM in particular) are just like any other ANN structure. Theoretically, a 1-hidden layer network can do any computation of a "deeper" network. ANN is a universal approximate of math functions. Still, multi-layer ANNs typically work better on more complex problems. Multi-layer typically needs less total connections, learns better and is less resource-demanding.
In particular, multi-layer LSTMs are believed to be better at determining complex in-time patterns. I think there is no rigorous proof for this, however.
Also, in practical applications - I did not see much improvement in network capabilities by adding additional LSTM layers. Adding more dense layers before/ after LSTM seemed to have a much better effect.