I'm starting to study RNN for a project of video prediction, but I encounter these CNN-LSTM models. Initially, I thought that is another name for RNN, but I think I get it wrong. Since I'm a beginner can someone explain me the difference between them?
1 Answer
An LSTM is a specific type of RNN. So let's just focus on the CNN part in CNN-LSTM.
What's the difference between a plain RNN and a CNN-RNN, (more generally called convolutional RNN or ConvRNN)?
The equations which define a vanilla RNN are (I'm omitting a bias term for clarity):
$h_t = \sigma_h(W_{hx}x_t + W_{hh}h_{t-1})$
$y_t = \sigma_y(W_{yh}h_t)$
where
- $h_t \in \mathbb{R}^m$ is the hidden state
- $x_t \in \mathbb{R}^n$ the input sequence (e.g. a string of characters)
- $y_t \in \mathbb{R}^l$ is the output vector
- $W_{hh} \in \mathbb{R}^{m\times m}$, $W_{hx} \in \mathbb{R}^{m\times n}$ and $W_{yh} \in \mathbb{R}^{l\times m}$ are matrices of parameters
- $\sigma_h$, and $\sigma_y$ are activation functions (such as the ReLu)
Now, in an CNN-RNN, the parameter matrices $W_{hh}$ and $W_{hx}$ are convolution matrices. We use them for input sequences which are typically better handled by convolutional neural networks, such as a sequence of images.