7
$\begingroup$

Is the LSTM-Architecture a subcategory of RNNs? Or are they totally different?

Literature doesn't seem to be unitary on this. This figure appears to explain the models to be alternatives, but I thought of them otherwise (LSTM to be a subcategory of RNN)

LSTM as a subcategory of RNN is mentioned in the Wikipedia article on LSTMs:

Long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture...

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
10
$\begingroup$

The Wikipedia article is more technically correct, in that the term RNN is formally taken to mean "a neural network with recurrent connections", and that includes many architectures that match this description, including LSTMs.

However, it is also common to see "RNN" used as a short-hand for a kind of "Vanilla RNN" or "basic RNN", where one or more layers have weights connecting the layer to itself (its own activations from $t-1$ are concatenated to the external inputs at $t$), and there are no other gates or special combinations, just those recurrent connections.

Oddly, this basic layer-based RNN archtecture is not listed in all the options on the Wikipedia page on RNNs - probably the closest are Elman networks and Jordan networks which are ways to implement the recurrent connection. It is a valid architecture choice, and can be effective. The LSTM and GRU architectures improve on it in terms of handling longer sequences and preserving important signals over them when training (e.g. matching a starting and ending quote in text processing).

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ So, I would clarify that this is a matter of terminology. If you define RNN (a short for recurrent neural networks) as the set of all neural networks with recurrent connections, then LSTMs could be considered a subset of RNN. However, if you define RNN as the vanilla RNN, then LSTMs are not a subset of RNNs (because vanilla RNNs do not have gates): it probably would be more the other way around (it may be possible to represent vanilla RNNs as LSTMs, though I don't remember the details now, so I am not really sure if this is possible). $\endgroup$
    – nbro
    Sep 13, 2021 at 16:19
  • $\begingroup$ @nbro: Yes. I think the looser terminology is very common, because for many problems "basic" RNNs and LSTMs/GRUs are interchangeable - in the sense that they can have same inputs and outputs, so using LSTM or GRU is a hyperparameter choice. For some reason that is phrased as using one of (RNN, GRU, LSTM) to solve a problem as opposed to one of (Elman, GRU, LSTM). I am not sure why $\endgroup$
    – Neil Slater
    Sep 13, 2021 at 16:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .