I am training an RL agent, using PPO, on a time-series environment that comes from a tabular dataset. The possible scores during an episode goes from -1 to positive infinity (though realistically, I would never expect an agent to get an episode score higher than 2.5 or 3).
By the end of training (3 million time steps, ~1350 episodes), I can see that the agent has a "mean episode reward" of roughly 0.4 . For context, below is an image of the plotted of that mean episode reward over time.
So, then, my expectation is that when I take this trained model and run it on an environment consisting of the same data it was trained on, it should have a performance somewhere around 0.4 reward on average per episode.
However, this is not what I find. In my case, the fully trained agent's mean reward is -0.3 over 10 episodes (episode length equal to episode length during training) which collectively span the entirety of the training data.
Why is the fully trained agent, who was just reported to have +0.4 mean score per episode, performing so poorly when run manually through a for-loop on the same training data?
By "manually through a for-loop", I mean that I have a for-loop through which I feed the agent observations one at a time, call
model.predict() to get an action and then enact that action etc.