2
$\begingroup$

I am working on a project for price movement forecasting and I am stuck with poor quality predictions.

At every time-step I am using an LSTM to predict the next 10 time-steps. The input is the sequence of the last 45-60 observations. I tested several different ideas, but they all seems to give similar results. The model is trained to minimize MSE.

For each idea I tried a model predicting 1 step at a time where each prediction is fed back as an input for the next prediction, and a model directly predicting the next 10 steps(multiple outputs). For each idea I also tried using as input just the moving average of the previous prices, and extending the input to input the order book at those time-steps. Each time-step corresponds to a second.

These are the results so far:

1- The first attempt was using as input the moving average of the last N steps, and predict the moving average of the next 10. At time t, I use the ground truth value of the price and use the model to predict t+1....t+10

This is the result

Predicting moving average

On closer inspection we can see what's going wrong:

Prediction seems to be a flat line. Does not care much about the input data.

2) The second attempt was trying to predict differences, instead of simply the price movement. The input this time instead of simply being X[t] (where X is my input matrix) would be X[t]-X[t-1]. This did not really help. The plot this time looks like this:

Predicting differences

But on close inspection, when plotting the differences, the predictions are always basically 0.

Plot of differences

At this point, I am stuck here and running our of ideas to try. I was hoping someone with more experience in this type of data could point me in the right direction.

Am I using the right objective to train the model? Are there any details when dealing with this type of data that I am missing? Are there any "tricks" to prevent your model from always predicting similar values to what it last saw? (They do incur in low error, but they become meaningless at that point).

At least just a hint on where to dig for further info would be highly appreciated.

Thanks!

UPDATE

Here is my config

{
    "data": {
        "sequence_length":30,
        "train_test_split": 0.85,
        "normalise": false,
        "num_steps": 5
    },
    "training": {
        "epochs":200,
        "batch_size": 64
    },
    "model": {
        "loss": "mse",
        "optimizer": "adam",
        "layers": [
            {
                "type": "lstm",
                "neurons": 51,
                "input_timesteps": 30,
                "input_dim": 101,
                "return_seq": true,
                "activation": "relu"
            },
            {
                "type": "dropout",
                "rate": 0.1
            },
            {
                "type": "lstm",
                "neurons": 51,
                "activation": "relu",
                "return_seq": false
            },
            {
                "type": "dropout",
                "rate": 0.1
            },
            {
                "type": "dense",
                "neurons": 101,
                "activation": "relu"
            },
            {
                "type": "dense",
                "neurons": 101,
                "activation": "linear"
            }
        ]
    }
}

Notice the last layer with 101 neurons. It is not an error. We just want to predict the features as well as the price. In other words, we want to predict the price for time t+1 and use the features predicted to predict the price and new features at time t+2, ...

$\endgroup$
  • 1
    $\begingroup$ What is your architecture? Knowing that it is an LSTM is not enough. What values are you using for the hyperparameters (e.g. number of epochs, mini-batch, number of neurons, number of layers, etc.)? Have you done a sanity check with simple data like a sine wave? Are you normalizing the data? If so, have you tried multiple methods? Why are you looking at one second interval data? Financial data is extremely noisy at that time scale even when averaged over 10 seconds. Why not consider one minute interval data or higher to at least prove your approach? $\endgroup$ – Brian O'Donnell Nov 3 '18 at 23:48
  • 1
    $\begingroup$ With your update the most obvious thing I would change is to normalize the data. You state that you are using price, moving average, and price difference but your configuration has 101 inputs. Why is that? Your input time steps is 30 but you state you are using 45-60 observations and the configuration says the sequence length is 30. Your sequence length should be greater than your input time steps by the number of steps you are predicting ahead. Why does your output layer have 101 neurons? Another useful piece of information would be a plot of the model's loss versus epoch. $\endgroup$ – Brian O'Donnell Nov 4 '18 at 15:17
1
$\begingroup$

Given a model that takes in a price and a second value, such as a moving average of the price, the following configuration is my recommendation. This is based on training on a history of 45 input time steps and forecasting out 10 steps in the future. I have assumed that your 'num_steps' is a stride through the training data. Note that I have not tested this and you will most likely need to tweak a parameter or two:

{
    "data": {
        "sequence_length": 55,
        "train_test_split": 0.85,
        "normalise": true,
        "num_steps": 5
    },
    "training": {
        "epochs":200,
        "batch_size": 64
    },
    "model": {
        "loss": "mse",
        "optimizer": "adam",
        "layers": [
            {
                "type": "lstm",
                "neurons": 100,
                "input_timesteps": 45,
                "input_dim": 2,
                "return_seq": true,
                "activation": "relu"
            },
            {
                "type": "dropout",
                "rate": 0.2
            },
            {
                "type": "lstm",
                "neurons": 100,
                "activation": "relu",
                "return_seq": false
            },
            {
                "type": "dropout",
                "rate": 0.2
            },
            {
                "type": "dense",
                "neurons": 1,
                "activation": "linear"
            }
        ]
    }
}

Try this out and plot your model's loss and accuracy by epoch. See Display Deep Learning Model Training History in Keras for some information and code in this area. Let me know if you are not using Keras and I will point you to another article/code solution.

$\endgroup$
  • $\begingroup$ I was wondering something a minute ago. Maybe there is a mistake in the code, but it is unclear to me. If we had applied regression on the price difference than the "Plot of differences" plot would be better, right? $\endgroup$ – user1050421 Nov 4 '18 at 16:32
  • $\begingroup$ If you mean using the regression line instead of the price differences my thought is that the performance would not be as good. It is a good idea to test out. Maybe also feed the regression line as a third value (i.e. price, difference, regression line). It is always good to try different data. $\endgroup$ – Brian O'Donnell Nov 4 '18 at 16:42
  • $\begingroup$ { "type": "dense", "neurons": 1, "activation": "linear" } I noticed you changed that part, but we wanted to predict the price as well as the features for t+2. So at time t+2, I reused the predicted features and the prices from time t+1 to predict the price for time t+2 and predict the features to be used again at time t+3. $\endgroup$ – user1050421 Nov 4 '18 at 17:22
0
$\begingroup$

Remember that any machine learning model works good only when there is a "rule" or a correlation between modeling data and modeled data. When there is not, even the best algorithm will not predict/classify correctly. I am not saying that this must be the case, but probably you have come pretty close to the physical limit of what can be achieved using this data.

Maybe you should consider adding or extracting new features to your model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.