I'm trying to train a PPO agent in a 3D balance ball environment. My action space is continuous.

In the following graph, each dot shows the average reward from 100 episodes.

enter image description here

Could this graph indicate that it's stuck at a local maximum? Do I need to promote exploring by increasing the entropy, or does this look like a bug with my implementation?

I am trying to maximize the average rewards.

When I look at the loss, it does seem like it is minimizing the cost. I thought that the loss function is dependent on the rewards because the advantage is calculated based on them. The normalized advantage is then factored into the policy loss, which should say which direction for the policy to step toward.

It seems like the actions are a bit noisy but the platform sometimes seems to try to keep the ball from falling off.


  • Learning rate = 0.01
  • Entropy Coefficient = 0.01
  • Value Function Loss Coefficient = 0.5
  • Gamma/Discount Factor = 0.995
  • MiniBatch Size = 512
  • Epochs = 3
  • Clip Epsilon = 0.1

1 Answer 1


I had the same problem where the reward kept decreasing and started to search for answers in the forum.

I let the model trained while I search. As the model trained, the reward started to increase. You can see the tensorboard graph for rewards in validation time.

Tensorboard graph for validation rewards.

The fall continued until around 100k~ steps and did not change a lot for 250k~ steps. After 350k~ th step, it slowly started to increase. Without knowing your number of steps trained, I would suggest training for more steps.

Also, I read about this (Reward first decreasing and then increasing) in an RL paper, if I find it I will mention it here.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .