What is the advantage of adding CNN to LSTM for forecasting sequential data?

Question

I am working with simulated sequential data and the goal is to forecast that data. Long-short-term-memory (LSTM) is one of the most advanced models to forecast time series according to this post. I can imagine that it is a good model because of the memory-cells they use which are useful when learning of the past.

This paper discussed the use of CNN in time-series analysis. It says:

CNN is suitable for forecasting time-series because it offers dilated convolutions, in which filters can be used to compute dilations between cells. The size of the space between each cell allows the neural network to understand better the relationships between the different observations in the time-series [14].

It even outperformed LSTM:

A specific architecture of CNN, WaveNet, outperformed LSTM and the other methods in forecasting financial time-series [16].

I see more and more posts about the usage of CNN in combination with LSTM, but I can't find any information about the advantages and disadvantages of using these in combination.

This post (Advantages of CNN vs. LSTM for sequence data like text or log-files), it is asked about the advantages of CNN vs. LSTM. But I would like to know the advantages and disadvantages of adding CNN to LSTM for forecasting univariate sequential data? Or should you use one of the two algorithms?

Answer 1

If your data are 2D + time then you might want to use something like ConvLSTM.

If you only care about 1D + time then you don't need to add CNN to LSTM you only use one or the other. In terms of pros and cons have a look at this empirical study on how dilated convolutions compare to LSTMs for modeling sequential data. If you're also interested in the more theoretical aspects, this paper shows how temporal convolutional networks are related to truncated RNNs.

Answer 2

A convolution layer can extract local patterns from its input (from the previous layer). Neighborhood size of the extracted pattern is relative to the kernel size. As we go deeper in the network the reception field of the layers increase. Accordingly, layers can recognize patters from larger neighborhoods of the input data. However, this increase in the reception field is very gradual. So, to understand patterns concerning large neighborhoods of input data, you need very deep networks. To ease this requirement, you can use dilated convolution layers, as in [1,2]. Hence, with shallower networks u can have reception fields as large as the whole input sequence. notice that many time series problems require patterns concerning the whole input sequence.
using LSTM layers in combination with convolution layers is also beneficial in many cases. in this scenario, several blocks of convolution layers first extract local hierarchical patterns. So the input sequence is represented by a reach collection of local patterns. Then, this patterns are feed to an LSTM to understand dependencies related to whole sequence. also, notice that an lstm layer generally requires more parameters to learn.
[1] Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P. A. (2019). Deep learning for time series classification: a review. Data mining and knowledge discovery, 33(4), 917-963.
[2] Wang, Z., Yan, W., & Oates, T. (2017, May). Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (IJCNN) (pp. 1578-1585). IEEE.