1
$\begingroup$

I am training an RL agent, using PPO, on a time-series environment that comes from a tabular dataset. The possible scores during an episode goes from -1 to positive infinity (though realistically, I would never expect an agent to get an episode score higher than 2.5 or 3).

By the end of training (3 million time steps, ~1350 episodes), I can see that the agent has a "mean episode reward" of roughly 0.4 . For context, below is an image of the plotted of that mean episode reward over time.

enter image description here

So, then, my expectation is that when I take this trained model and run it on an environment consisting of the same data it was trained on, it should have a performance somewhere around 0.4 reward on average per episode.

However, this is not what I find. In my case, the fully trained agent's mean reward is -0.3 over 10 episodes (episode length equal to episode length during training) which collectively span the entirety of the training data.

Why is the fully trained agent, who was just reported to have +0.4 mean score per episode, performing so poorly when run manually through a for-loop on the same training data?

By "manually through a for-loop", I mean that I have a for-loop through which I feed the agent observations one at a time, call model.predict() to get an action and then enact that action etc.

Improve this question
$\endgroup$
5
  • $\begingroup$ You are right, those two numbers should match. To me, it sounds like some unexpected behavior in the code, so I would carefully go through the code again and again. I don't think someone can identify what's wrong based on your description. $\endgroup$
    – Chillston
    Jun 29, 2022 at 23:22
  • 1
    $\begingroup$ @Chillston I see. Thank you for your input. I'm pretty new to reinforcement learning, so at this stage, it's just hard for me to judge what's common/typical behavior vs. what's an error. I should note that my environment comes from a tabular time series dataset, and on every episode during training, I randomly choose a "starting point" from the dataset. In "testing", I naturally do not choose the starting points randomly, but chronologically. So, maybe this randomness is the cause of the issue (maybe I "happen" to choose "bad" start points)? $\endgroup$
    – Vladimir Belik
    Jun 30, 2022 at 0:25
  • $\begingroup$ @Chillston Perhaps I will run a few thousand episodes in "test" mode, randomly selecting a start point each time, to see if there's a difference between the average "test" result vs. the mean episode reward during training (to negate the effect of potentially unfortunate choice of starting points). Should I add some of my code to the question? $\endgroup$
    – Vladimir Belik
    Jun 30, 2022 at 0:27
  • 1
    $\begingroup$ @Chillston It appears that that's the issue - when I run a bunch of trials with random starting points, the mean "manual" score is about the same as the mean episode reward. $\endgroup$
    – Vladimir Belik
    Jun 30, 2022 at 20:54
  • 1
    $\begingroup$ Very nice, that makes a lot of sense! :) Regarding the addition of code - in this case the question is more about debugging, this you should post on StackOverflow. But you sorted it out already anyways. $\endgroup$
    – Chillston
    Jul 1, 2022 at 17:44

1 Answer 1

Reset to default
1
$\begingroup$

I have found the issue.

Essentially, what appears to be happening is when I do my (only) 10 manual runs, I happen to get a bad sample of starting points.

During training, for each episode, I randomly select a start point in the data so that the agent doesn't overfit to a single starting state. However, during my "manual testing", I just chose 10 chronological starting points to run episodes from. It appears that the 10 chronological starting points just happen to be inconvenient/bad, so the results aren't great.

To verify this idea, I ran 500 "manual for-loop tests" of the fully trained model by randomly selecting a starting point each time (just like I did during training). When I did this, the mean result across the 500 runs was quite similar to the final "episode mean reward" of my agent during training.

So, it appears to come down to the "luck of the draw" when it comes to the episode starting points.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .