I have an issue with the normalization of the database (a large time series) for my DQN. I obtained optimal results and saved the NN (5 LSTM layers) weights training on a database normalized as such: I divided it into consecutive batches of 96 steps (the window size that my NN gets as input) and I normalized each batch respectively with Z-score. However, I am unable to extend these results to an online setting, as online I only have access to the last 96 elements, and thus I can only normalize according to the last 96. This small difference actually causes a sharp decrease in the performance of my DQN, as the weights of the NN were perfectly tuned for the first normalization but are not great with the online normalized database. In a nutshell, the problem is that only every 96 steps the first normalized database and the online one are the same, for all steps in between this is not happening. I have the weights for the first one, but I cannot find a way to exploit them for the online one.
What I have tried so far with the online database:
If I normalize every last 96 steps, and act for every new step (as it should be), the performances are quite bad.
If I normalize every last 96 steps, and act just every 96 steps (repeating the same action in between), the agent is actually picking the optimal action every 96 steps (like in the offline setting), so the results are somewhat decent but far from optimal for the long period between the actions. If I try with shorter periods, like 48, performances decrease sharply as it only acts optimally every 2 actions.
I don't know if there is a way to tune the optimal weights for the online database, acting directly on them without going through training again. It would be nice to understand why the NN picks its actions at each step in the optimal setting, so that I would be able to follow its strategy, but I'm not aware if it's possible to actually deduct this from the analysis of weights and features, especially for a multi-layer LSTM network. Otherwise, I was thinking about something like normalizing the online database directly through similarities with the old batches of 96 (using their mean and std) or something like that. Anything that would help reducing the time between optimal actions to around 50-60 steps instead of 96 would be enough to provide a nearly optimal strategy, so at this point, I would consider any kind of (unelegant) method to get what I want.
I don't know if any of these is feasible, but retraining the agent is very difficult as every single time but once the agent got stuck in suboptimal strategies, this is why I am trying to get around this problem using the optimal weights I have instead of retraining.