What is the difference between CNN-LSTM and RNN?

Question

I'm starting to study RNN for a project of video prediction, but I encounter these CNN-LSTM models. Initially, I thought that is another name for RNN, but I think I get it wrong. Since I'm a beginner can someone explain me the difference between them?

Answer 1

An LSTM is a specific type of RNN. So let's just focus on the CNN part in CNN-LSTM.

What's the difference between a plain RNN and a CNN-RNN, (more generally called convolutional RNN or ConvRNN)?

The equations which define a vanilla RNN are (I'm omitting a bias term for clarity):

$h_t = \sigma_h(W_{hx}x_t + W_{hh}h_{t-1})$

$y_t = \sigma_y(W_{yh}h_t)$

where

• $h_t \in \mathbb{R}^m$ is the hidden state
• $x_t \in \mathbb{R}^n$ the input sequence (e.g. a string of characters)
• $y_t \in \mathbb{R}^l$ is the output vector
• $W_{hh} \in \mathbb{R}^{m\times m}$, $W_{hx} \in \mathbb{R}^{m\times n}$ and $W_{yh} \in \mathbb{R}^{l\times m}$ are matrices of parameters
• $\sigma_h$, and $\sigma_y$ are activation functions (such as the ReLu)

Now, in an CNN-RNN, the parameter matrices $W_{hh}$ and $W_{hx}$ are convolution matrices. We use them for input sequences which are typically better handled by convolutional neural networks, such as a sequence of images.