Structure of LSTM RNNs

Question

I have some very basic questions here. This is probably because I didn't read the relevant documents closely enough. If I used some terminology incorrectly, please point them out. Thank you!

1. For units/neurons in the hidden layers, they are referred to as memory blocks and each memory block can contain multiple memory cells?

2. And one memory cell looks like these two?

3. For units/neurons in the input and output layers, they are simply regular neurons? Like those in a feedforward network?

4. Regarding the network structure, the structure of an RNN essentially looks like that of a feedforward network, except the neurons in the hidden layer have been replaced by neurons with recurrent connections. Is this an accurate description?

5. LSTM RNNs then replaces those neurons in the hidden layers with memory blocks (which contain memory cells that have recurrent connections). Is this an accurate description?

Answer 1

Structure of LSTM Networks

[Are] units/neurons in the hidden layers ... referred to as memory blocks and each memory block can contain multiple memory cells?

Although an early guess at the way biological neurons work inspired the Perceptron design that later became multilayer (MLP) and then recurrent (RNN), these artificial network building blocks are not very much like biological neurons. The term is still used by some, but, because of this increased interdisciplinary awareness, the more common terms are cell, unit, or block.

These last two terms can be a little ambiguous when entire networks are used as building blocks in larger AI architectures. This is not a show stopper and also occurs in other engineering disciplines where sub-circuit blocks are components in more complex blocks, and sub-assemblies can be parts of larger mechanical assemblies.

And one memory cell looks like these two?

The images in the questions aren't clear representations. This visualization of an LSTM cell¹ shows the attenuation parameters, each labeled with a capital W.

These LSTM cells are arranged as a layer and each pair of adjacent cells within the same layer are connected $(C_t, \, h_t) \rightarrow (C_{t+1}, \, h_{t+1})$. The input in this articular diagram is $x_t$ and the output to the next layer is $h_t$.

For units in the input and output layers, they are simply regular neurons, like those in a feed forward network?

They can be, but there are many topological designs in use that use LSTM cells in some layers and Perceptrons in others, along with side-by-side networks, bi-directional designs, recursive topologies, and completely different cell types, such as the very popular GRU cell.

[Is] the structure of an RNN essentially like that of a feed-forward network, except the neurons in the hidden layer have been replaced by neurons with recurrent connections ...?

LSTM RNNs then replaces those neurons in the hidden layers with memory blocks (which contain memory cells that have recurrent connections) ...?

No general rule can be assumed, especially since new designs emerge nearly every week, and significant topological advancements occur about once every one or two years. This is an example² of a topology organized to support a specific processing use case.

---

References

[1] Visualizations of RNN units — Diagrams of RNN unrolling, LSTM and GRU, by Jakub Kvita, 2016, and inspired by Oinkina

[2] Automatic Image Captioning using Deep Learning (CNN and LSTM), Faizan Shaikh, 2018

Answer 2

1. For units/neurons in the hidden layers, they are referred to as memory blocks and each memory block can contain multiple memory cells?

yes each lstm unit contains multiple memory cells.

2. And one memory cell looks like these two?

   

in the diagram each cell is one lstm unit , this did not show forget gates though.

3.For units/neurons in the input and output layers, they are simply regular neurons? Like those in a feedforward network?

yes they are regular feed forward connected.

4.*Regarding the network structure, the structure of an RNN essentially looks like that of a feedforward network, except the neurons in the hidden layer have been replaced by neurons with recurrent connections. Is this an accurate description? *

yeah but this is for vanilla rnn in lstm/gru variants gating mechanism is introduced.

5 *LSTM RNNs then replaces those neurons in the hidden layers with memory blocks (which contain memory cells that have recurrent connections). Is this an accurate description? *

the main motivation behind this is to eradicate something known as "vanishing/exploding" gradient problem . which tends to either make the gradients explode or make them zero as the gradients flow through the layers.