I've written a Double DQN-based stock trading bot using mainly time series stock data.
I've recently upgraded my Experience Replay(ER) code with a version of Prioritized Experience Replay (PER) similar to the one written by OpenAI. My DQN's reward function is the stock return over 30 days (the length of my test window).
The strange thing is, once the bot has been trained using the same set of time series data and let free to trade on unseen stock data, the version that uses PER actually comes up with worse stock returns than the version using a regular ER.
This is not quite what I'd expected but it's very hard to debug and see what might have gone wrong.
So my question is, will PER always perform better than a regular ER? If not, when/why not?