# How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?

I am currently using the PPO method from the Nvidia's Isaac gym to train an agent for my robot. Below, you can see the plot which corresponds to a training process.

I know that something is massively wrong with my training process (since the robot does not manage to get a nice policy), so I am trying to understand the training process more with the help of the below values being logged during the problematic training.  So far, I could find out what this value function means: the reward function stabilizes, thus the loss value function also stabilizes, which means that my robot should explore more to fail and learn from the fail!

But what about the other two plots, surrogate and mean_noise_std? How should one interpret those values?

The ideal training process

Loss. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some huge-dimensional parameter space. For the Reinforcement Learning problems, the optimal point is supposed to maximize the expected return - while most of the Deep Learning classification problems are minimizing some loss.

Most of the modern Deep Learning programming frameworks was created with the classification problems in mind, so they call the optimization goals "losses": tf.keras, pytorch. The good news is that you can easily convert return-maximization problem to loss-minimization problem by just changing the sign of your target function.

Given all the above, the fact that your loss is going up instead of going down doesn't seem to be a good sign.

Value Function. Given an agent's state, Reinforcement Learning tries to find the best agent's action that maximized the expected return. The problem is that the expected return is also unknown. Note on the difference between "reward" and "return". Rewards $$r_t$$ are immediate bits of positive/negative reinforcement that the agent receives obtain after taking an action. The "return" $$G_t$$ is the function of a particular trajectory - it is the discounted sum of future rewards starting from step $$t$$:

$$G_t = \sum_k \gamma^k r_{t+k+1}$$

The expectation of the return at state $$s$$ given a policy $$\pi$$ is called value $$V^\pi(s)$$:

$$V^\pi(s) = \mathbb{E}_{a\sim\pi}\left[ G_t\right|\left. s_t=s\right]$$

In principle, it is possible to estimate $$V^\pi(s)$$ by doing Monte Carlo: start from the state $$s$$ and run many simulations following $$\pi$$ and average all the returns you've got. For most practical cases this approach is infeasible. Modern Deep Reinforcement Learning algorithms try to reduce computational load by using neural networks to approximate $$V(s)$$.

Having the value function approximation $$V(s)$$, one can then try to optimize the policy approximation function $$\pi(a|s)$$ using the policy gradient update. Such setup: with one approximator for policy and another approximator for value function is called an "Actor-Critic" family of Reinforcement Learning algorithms.

BTW, it doesn't look like your value function "stabilizes" during training. And it also doesn't change too much if you look at the vertical scale of your plot.

Surrogate Loss. In practice, the policy gradient optimization step above suffers from instabilities. The gradient step tends to change the policy too strongly, which makes the approximation error of the value function too large.

To deal with this, the TRPO/PPO algorithms introduce a kind of clipping of the loss gradient that constrains the policy gradient changes from being too large. The new function that produces these clipped/constrained gradients is called a "surrogate loss".

mean_noise_std. This doesn't look like a standard name for anything, but I suspect that this is a version of PPO with artificial noise added for better exploration of the agent.

• Thank you for the kind explanation. "The problem is that the expected reward is also unknown" - why is this the case? We have physic simulator. We can just simulate the robot's behavior with the given agent's state.
– Joe
May 15 at 16:35
• "Given all the above, the fact that your loss is going up instead of going down doesn't seem to be a good sign." - I have added plots of an ideal training process. As you can see, as the reward gets maximized around 1k iteration, the loss also gets maximized. This behavior somehow doesn't fit your explanation... Could you elaborate a bit more?
– Joe
May 15 at 16:39
• @Joe First of all, I've made a slight terminological mistake mixing up "reward" and "return". I've tried to expand on your first comment - In principle, you can estimate $V(s)$ by running simulations of the robot's behavior from the given agent's state. Say you choose to run it 100 times per state. And you have 10000 trajectories. And each trajectory has 10000 states in it.... May 15 at 19:37
• @Joe I cannot comment on your "ideal training"plots apart from the fact that you have "reward" here instead of "loss". I'll need more information on how you got it and why is it "ideal". Are you trying to do some online course and that's an expected behavior? May 15 at 19:40
• I am using Nvidia Isaac Gym and the later plots are from a given example from Nvidia thats why I called it ideal. In the example, the robot also manages to reach the goal position about after 1k iteration.
– Joe
May 15 at 20:03