Price difference predictions curve almost vanished

With a team, we are studying how it is possible to predict the price movement with high-frequency. Instead of predicting the price directly, we have decided to try predicting price difference as well as the features. In other words, at time t+1, we predict the price difference and the features for time t+2. We use the predicted features from time t+1 to predict the price at time t+2.

We got very excited, because we thought getting good results with the following graph

We got problems in production and we wasn't known the problem till we plot the price difference.

Here is the content of the config file

{
"data": {
"sequence_length":30,
"train_test_split": 0.85,
"normalise": false,
"num_steps": 5
},
"training": {
"epochs":200,
"batch_size": 64
},
"model": {
"loss": "mse",
"layers": [
{
"type": "lstm",
"neurons": 51,
"input_timesteps": 30,
"input_dim": 101,
"return_seq": true,
"activation": "relu"
},
{
"type": "dropout",
"rate": 0.1
},
{
"type": "lstm",
"neurons": 51,
"activation": "relu",
"return_seq": false
},
{
"type": "dropout",
"rate": 0.1
},
{
"type": "dense",
"neurons": 101,
"activation": "relu"
},
{
"type": "dense",
"neurons": 101,
"activation": "linear"
}
]
}
}


Prices don't change very fast. Therefore, the next price is almost always very close to the last price. In other words, P_{t+1} - P_{t} is very often close to zero or zero directly. If there is too many zeros then the network will only recognize the zeros. The model has picked up on that.

I guess the model learned almost nothing except the very simple relationship that the next price is close to the last price. There is not necessarily anything wrong with the model. Predicting stock prices should be a very hard problem.

So a straightforward improvement should be of taking the features as a whole instead of their difference.

I want to keep working with price difference instead of the price in itself because we are making the series potential more stationary.

What might be a good solution to deal with the repetitive zeros related to our "price difference" problem? Does applying the log-return is a better idea than applying price differences?

Does a zero inflated estimators is a good idea? First predict whether it's gonna be a zero. If not predict the value. https://gist.github.com/fonnesbeck/874808 ?