1
$\begingroup$

In time Series prediction, we have a stream of vectors. There are different approaches for accounting for the temporal patterns between these vectors.

There's two that I'm considering. An LSTM or augmenting the feature space. What's the difference between the two? The most obvious to me is that an LSTM is more expressive and can get superior accuracy if modelled properly.

$\endgroup$
1
$\begingroup$

LSTM is a neural network which learns for an input x an output y. In additional to CNNs or MLPs it considers a hiddenstate h (which is influenced by prvious inputs) when your next input x is feed into the network.

Augmenting the feature Space is a technique which you do previous training your LSTM (to augment your data set in order to generatre more data and let the LSTM better generalize to new data). In the field of image recognition you can rotate your images by 40 degree to generate a new one. This process is known as data augmentation. Such methods also appliable to time series.

In summary: first, you start with augmenting your input feature space in order to improve prediction accuracy and then training your LSTM with the augmeneted training data set.

$\endgroup$
  • $\begingroup$ I don't think this answers the question. $\endgroup$ – echo Apr 9 '18 at 19:20
  • 1
    $\begingroup$ This answer first distinguish and relates the two terms LSTM and feature space augmentation. For a more detailed answer please give more information (for instance a paper to each term). $\endgroup$ – user3352632 Apr 10 '18 at 7:16
  • 1
    $\begingroup$ Everything @user3352632 says is correct and makes sense though :p $\endgroup$ – Andreas Storvik Strauman Apr 11 '18 at 7:01
0
$\begingroup$

I just read this in a recent Bengio paper and it's pretty obvious. He says that there are zero differences between a short-term memory and an augmented feature space. However, if you want to capture long-term dependencies without blowing up the feature space, you'd want to use an LSTM because traditional approaches can't dynamically learn what to "remember".

$\endgroup$
  • $\begingroup$ Correct! However, the Elman-RNN (vanilla) can in theory learn long term dependencies as well as an LSTM, as Bengio also has a paper on. The problem is that it is very sensitive to the hyper parameters and thus they are very hard to tune. $\endgroup$ – Andreas Storvik Strauman Apr 8 '18 at 21:12
  • $\begingroup$ No, wait what? What do you mean by "augmented feature space"? $\endgroup$ – Andreas Storvik Strauman Apr 8 '18 at 21:13
  • $\begingroup$ Vector concatenation: <Vector T>.<Vector T-1>.<Vector T-2> $\endgroup$ – echo Apr 8 '18 at 21:49
  • 1
    $\begingroup$ No. Maybe. I don't understand how you can compare concatenating vectors and LSTM? $\endgroup$ – Andreas Storvik Strauman Apr 11 '18 at 6:02
  • 1
    $\begingroup$ By LSTM you mean long short term memory recurrent neural network, right? $\endgroup$ – Andreas Storvik Strauman Apr 11 '18 at 7:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.