What's the difference between LSTM and RNN? I know that RNN is a network layer used in neural networks, but what exactly is an LSTM? Is it also a network layer with the same characteristics?


You can describe a recurrent neural network (RNN) or a long short-term memory (LSTM), depending on the context, at different levels of abstraction. For example, you could say that an RNN is any neural network that contains one or more recurrent (or cyclic) connections. Or you could say that layer $l$ of neural network $N$ is a recurrent layer, given that it contains units (or neurons) with recurrent connections, but $N$ may not contain only recurrent layers (for example, it may also be composed of feedforward layers, i.e. layers with units that contain only feedforward connections).

In any case, a recurrent neural network is almost always described as a neural network (NN) and not as a layer (this should also be obvious from the name). On the other hand, an LSTM can refer to an LSTM unit (or neuron), an LSTM layer (many LSTM units) or an LSTM neural network (an NN with LSTM units or layers), depending on the context.

An LSTM unit is a recurrent unit, that is, a unit (or neuron) that contains cyclic connections, so an LSTM network is a recurrent network. The main difference between an LSTM unit and a standard RNN unit is that the LSTM unit is more sophisticated. More precisely, it is composed of the so-called gates that supposedly regulate better the flow of information through the unit.

Here's a typical representation (or diagram) of an LSTM (more precisely, an LSTM with a so-called peephole connection).

enter image description here

This can actually represent both an LSTM unit (and, in that case, the variables are scalars) or an LSTM layer (and, in that case, the variables are vectors or matrices). You can easily see from this diagram that an LSTM unit (or layer) is composed of gates and recurrent connections. It's also composed of a cell. To understand the details (i.e. the purpose of all these components, such as the gates), you should e.g. read the paper that originally proposed the LSTM by S. Hochreiter and J. Schmidhuber. However, there may be more accessible and understandable papers.

On the other hand, any recurrent neural network (either an LSTM or not) may be represented as a graph that contains one or more cyclic connections. For example, the following diagram may represent both a standard RNN or an LSTM network (or maybe a variant of it, e.g. the GRU).

enter image description here

To conclude, any recurrent network is particularly suited for tasks that involve sequences (because of the recurrent connections). For example, they are often used for machine translation, where the sequences are sentences or words. In practice, an LSTM is often used, as opposed to a vanilla (or standard) RNN, because it often works better. In fact, the LSTM was introduced to solve a problem that standard RNNs suffer from, i.e. the vanishing gradient problem.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.