enter image description here

I'm trying to train a PPO agent and my average rewards graph looks like this. Could this indicate that it's stuck at a local maximum? Do I need to promote exploring by increasing the entropy or does this look like a bug with my implementation? Also, my action space is continuous. Thanks!


Learning rate = 0.01

Entropy Coefficient = 0.01

Value Function Loss Coefficient = 0.5

Gamma/Discount Factor = 0.995

MiniBatch Size = 512

Epochs = 3

Clip Epsilon = 0.1

  • $\begingroup$ Is the graph the right way up? Are you running a variation where you want to minimise a cost(as opposed to maximise reward) and are plotting the cost? When you view the agent's behaviour, does it seem to be achieving any goals? $\endgroup$ May 9 '19 at 13:26
  • $\begingroup$ Yeah, the graph is accurate. It each dot shows the average reward from 100 episodes. When I look at the loss, it does seem like it is minimizing the cost. I thought that the loss function is dependent on the rewards because the advantage is calculated based on them. The normalized advantage is then factored into the policy loss, which should say which direction for the policy to step toward. My environment is the 3D ball balancing environment. It seems like the actions are a bit noisy but the platform sometimes seems to try to keep the ball from falling off $\endgroup$
    – Tony Ho
    May 9 '19 at 14:33
  • $\begingroup$ So I think there is a bit of learning happening $\endgroup$
    – Tony Ho
    May 9 '19 at 14:36
  • $\begingroup$ OK, so according to your graph the reward starts high and then gets lower . . . and you are trying to maximise reward? So the untrained agent has the best performance according to that graph? $\endgroup$ May 9 '19 at 16:03
  • $\begingroup$ Yes, that's correct. So this is most likely a bug in my implementation? $\endgroup$
    – Tony Ho
    May 9 '19 at 23:15

I had the same problem where the reward kept decreasing and started to search for answers in the forum.

I let the model trained while I search. As the model trained, the reward started to increase. You can see the tensorboard graph for rewards in validation time.

Tensorboard graph for validation rewards.

The fall continued until around 100k~ steps and did not change a lot for 250k~ steps. After 350k~ th step, it slowly started to increase. Without knowing your number of steps trained, I would suggest training for more steps.

Also, I read about this (Reward first decreasing and then increasing) in an RL paper, if I find it I will mention it here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.