# PPO when does the update happen?

In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value function and the value target, which I can only assume is the discounted sum of rewards that can only be obtained at the END of the episode?

So this might be a moment of stupidity for me, but

1. Is the value target in PPO set only at the end of the episode using the discounted sum of rewards? or is there a secret way of setting these value targets that I am missing?

2. If a learning update indeed takes place every learning step (before the end of the episode), then how does this TD-learning happen - does it use some other approximate of the value target?

Sincerely, a frustrated student

## Monte Carlo vs. Temporal Difference (TD)

Let's start with the distinction between these two. When you have a sequence of rewards observed from the environment and a neural network predicting the value of each state, then you can create target values that your predictions should move closer to in a couple of ways. You can look at the full episode and use the actual observed rewards with discounting to create your target, this is called the Monte Carlo estimation. This target value is an estimation of the value function from your initial state.

$$\widehat v_\pi(s_0) = y^{MC} = r_1 + \gamma r_2 + \gamma^2 r_3 + \cdot\cdot\cdot + \gamma^{T-1}r_T$$

Then, you update the NN parameters to get better at predicting the value of the state based on this estimate. Another way is to update your neural network sooner by only using a partial trajectory (of length $$k$$) and rely on your NN to estimate the rest of the trajectory.

$$\widehat v(s_0) = y^{TD} = r_1 + \gamma r_2 + \gamma^2 r_3 + \cdot\cdot\cdot + \gamma^{k-1} r_k + \gamma^k \widehat v(s_k)$$

The last term uses the NN to estimate the remaining discounted rewards from the episode without observing it.

## Proximal Policy Optimisation

You don't need to wait until the end of an episode to receive rewards. If you have access to intermediate rewards, then you can update the value network sooner. PPO uses the advantage function when calculating the objective (and the loss) which is also done similarly to the TD approach. Both the n-step and the Generalised Advantage Estimation relies on the NN to fill in some of the unobserved values.

Original paper on PPO gives a nice description of the algorithm (the version with clipping the probability ratio is probably easier to understand): Proximal Policy Optimization Algorithms (Schulman et al., 2017)

OpenAI has a good description of general policy gradient algorithms and PPO as well, it's worth checking out.

• I mean...yes, there are immediate rewards. But my question is actually what update does the PPO do at every step? What is changed? I don't even need the actual formula. Just need to know if it changes the weights of either the value function (critic) or policy network (actor) at any step before the end of the episode. And if yes, then what does it use to make the update? is value target = advantage? Sep 1, 2021 at 18:41
• I understand. But in order to get the gradients that will be used to nudge the weights, it needs to calculate a loss. One of the terms in the PPO loss is the value function loss. How does it calculate that without the value target - which can only be estimated at the end of the episode from the rewards? Does it use the discounted sum of rewards only until that step as the value target? Sep 1, 2021 at 19:53
• Okay, thanks! So my final takeaway is that it uses the rewards observed up until that point to create a 'value target' at every step from which we can obtain value function loss by calculating difference between value function estimate and that 'value target'. To calculate this partial value target in between the episode, it uses only the discounted sum of rewards seen until that step, correct? Sep 1, 2021 at 20:24
• Yes, and an estimation of the value function from the final step via the value network.
– user42664
Sep 1, 2021 at 20:30