What are some examples of LSTM architectures?

Question

I've been doing some class assignments recently on building various neural networks. For convolutional networks, there are several well-known architectures such as LeNet, VGG etc. Such "classic" models are frequently referenced as starting points when building new CNNs.

Are there similar examples for RNN/LSTM networks? All I've found so far are articles and slides explaining recurrent neurons, LSTM layers, and the math behind them, but no well-known examples of entire multi-layered network architectures, unlike CNNs which seem to have in abundance.

Answer 1

In the paper, LSTM: A Search Space Odyssey (2017), by Klaus Greff et al., eight LSTM variants on three representative tasks (speech recognition, handwriting recognition, and polyphonic music modeling) are compared.

The compared variants are

1. Vanilla LSTM features three gates (input, forget, output), block input, a single cell, an output activation function, and peephole connections (connections from the cell to the gates). The output of the block is recurrently connected back to the block input and all of the gates. The vanilla LSTM is trained using gradient descent and back-propagation through time (BPTT). The original LSTM (which is not the vanilla LSTM) does not contain, for example, the forget gate or the peephole connections (but the cell possesses a constant error carousel, a constant weight of $1$).

2. LSTM trained based on the decoupled extended Kalman filtering (DEKF-LSTM), which enables the LSTM to be trained on some pathological cases at the cost of high computational complexity.

3. Vanilla LSTM trained with an evolution-based method (called evolino), instead of BPTT.

4. LSTM block architectures evolved with a multi-objective evolutionary algorithm, so that to maximize fitness on context-sensitive grammar.

5. LSTM architectures for large scale acoustic modeling, which introduces a linear projection layer that projects the output of the LSTM layer down before recurrent and forward connections in order to reduce the number of parameters for LSTM networks with many blocks.

6. An LSTM architecture with a trainable scaling parameter for the slope of the gate activation functions, which improves the performance of LSTM on an offline handwriting recognition dataset.

7. Dynamic Cortex Memory, an LSTM composed of recurrent connections between the gates of a single block, but not between different blocks, which improves the convergence speed of LSTM.

8. Gated Recurrent Unit (GRU), which simplifies the architecture of the LSTM by combining the input and forget gate into an update gate.

There are other related neural network architectures, such as the neural Turing machine (NTM) or differentiable neural computer (DNC). In general, there are several architectures that use LSTM blocks, even though they are not just recurrent neural networks. Other examples are the neural programmer-interpreter (NPI) or the meta-controller.