I've written a Double DQN which can do either 1-step or multi-step learning. It also has a prioritised reply buffer. The internal network is an LSTM.
My input data is a series of time series data and other features derived from them.
I feed this model with a long set of time series data for training and let the trained model make its decisions on an independent test data set. I run this same test many times.
The observation is that the model doesn't give consistent performance according to my measurement metric. Sometimes it performs better than a benchmark, other times it gives out downright wrong actions.
I can only think of the following two possible causes of this problem:
The model has not been trained thoroughly enough. i.e. I haven't gone through enough number of episodes.
My current exploration epsilon is 0.1. It's too big and the model 'explores' too often.
I can test these theories, but, to tell the truth, I am not entirely convinced they are the reasons my model is not consistent.
I would like to hear some opinions and advice.