LSTM is supposed to be the right tool to capture path-dependency in time-series data.
I decided to run a simple experiment (simulation) to assess the extent to which LSTM is better able to understand path-dependency.
The setting is very simple. I just simulate a bunch (N=100) paths coming from 4 different data generating processes. Two of these processes represent a real increase and a real decrease, while the other two fake trends that eventually revert to zero.
The following plot shows the simulated paths for each category:
The candidate machine learning algorithm will be given the first 8 values of the path ( t in [1,8] ) and will be trained to predict the subsequent movement over the last 2 steps.
In other words:
the feature vector is
X = (p1, p2, p3, p4, p5, p6, p7, p8)
the target is
y = p10 - p8
I compared LSTM with a simple Random Forest model with 20 estimators. Here are the definitions and the training of the two models, using Keras and scikit-learn:
# LSTM model = Sequential() model.add(LSTM((1), batch_input_shape=(None, H, 1), return_sequences=True)) model.add(LSTM((1), return_sequences=False)) model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy']) history = model.fit(train_X_LS, train_y_LS, epochs=100, validation_data=(vali_X_LS, vali_y_LS), verbose=0)
# Random Forest RF = RandomForestRegressor(random_state=0, n_estimators=20) RF.fit(train_X_RF, train_y_RF);
The results are the summarized by the following scatter plots:
As you can see, the Random Forest model is clearly outperforming the LSTM. The latter seems to be not able to distinguish between the real and the fake trends.
Do you have any idea to explain why this is happening?
How would you modify the LSTM model to make it better at this problem?
- The data points are divided by 100 to make sure gradients do not explode
- I tried to increase the sample size, but I noticed no differences
- I tried to increase the number of epochs over which the LSTM is trained, but I noticed no differences (the loss becomes stagnant after a bunch of epochs)
- You can find the code I used to run the experiment here