LSTM is supposed to be the right tool to capture path-dependency in time-series data.

I decided to run a simple experiment (simulation) to assess the extent to which LSTM is better able to understand path-dependency.

The setting is very simple. I just simulate a bunch (N=100) paths coming from 4 different data generating processes. Two of these processes represent a real increase and a real decrease, while the other two fake trends that eventually revert to zero.

The following plot shows the simulated paths for each category:

Simulated paths

The candidate machine learning algorithm will be given the first 8 values of the path ( t in [1,8] ) and will be trained to predict the subsequent movement over the last 2 steps.

In other words:

  • the feature vector is X = (p1, p2, p3, p4, p5, p6, p7, p8)

  • the target is y = p10 - p8

I compared LSTM with a simple Random Forest model with 20 estimators. Here are the definitions and the training of the two models, using Keras and scikit-learn:

model = Sequential()
model.add(LSTM((1), batch_input_shape=(None, H, 1), return_sequences=True))
model.add(LSTM((1), return_sequences=False))
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])
history = model.fit(train_X_LS, train_y_LS, epochs=100, validation_data=(vali_X_LS, vali_y_LS), verbose=0)
# Random Forest
RF = RandomForestRegressor(random_state=0, n_estimators=20)
RF.fit(train_X_RF, train_y_RF);

The results are the summarized by the following scatter plots:

enter image description here

As you can see, the Random Forest model is clearly outperforming the LSTM. The latter seems to be not able to distinguish between the real and the fake trends.

Do you have any idea to explain why this is happening?

How would you modify the LSTM model to make it better at this problem?

Some remarks:

  • The data points are divided by 100 to make sure gradients do not explode
  • I tried to increase the sample size, but I noticed no differences
  • I tried to increase the number of epochs over which the LSTM is trained, but I noticed no differences (the loss becomes stagnant after a bunch of epochs)
  • You can find the code I used to run the experiment here
  • $\begingroup$ With my limited knowledge I belive you need to set a larger amount of neurons for the LSTM network as they are currently set to 1 which would produce a 1-dimensional space with limited learning capacity regardless of inputs / outputs required $\endgroup$ – benbyford Apr 9 '19 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.