Skip to main content
made answer more useful
Source Link
user42664
user42664

Updated response to include more information from the discussion.

Monte Carlo vs. Temporal Difference (TD)

Let's start with the distinction between these two. When you have a sequence of rewards observed from the environment and a neural network predicting the value of each state, then you can create target values that your predictions should move closer to in a couple of ways. You can look at the full episode and use the actual observed rewards with discounting to create your target, this is called the Monte Carlo estimation. This target value is an estimation of the value function from your initial state.

$$ \widehat v_\pi(s_0) = y^{MC} = r_1 + \gamma r_2 + \gamma^2 r_3 + \cdot\cdot\cdot + \gamma^{T-1}r_T $$

Then, you update the NN parameters to get better at predicting the value of the state based on this estimate. Another way is to update your neural network sooner by only using a partial trajectory (of length $k$) and rely on your NN to estimate the rest of the trajectory.

$$ \widehat v(s_0) = y^{TD} = r_1 + \gamma r_2 + \gamma^2 r_3 + \cdot\cdot\cdot + \gamma^{k-1} r_k + \gamma^k \widehat v(s_k) $$

The last term uses the NN to estimate the remaining discounted rewards from the episode without observing it.

Proximal Policy Optimisation

You don't need to wait until the end of an episode to receive rewards. At least, not in the general sense. If you have access to intermediate rewards, then you can update the value network sooner. But PPO actually uses the advantage function when calculating the objective (and the loss) which is also andone similarly to the TD approach. Both the n-step rollout or done viaand the Generalised Advantage Estimation which still relies on bootstrapping from estimates at different timestepsthe NN to fill in some of the unobserved values.

Original paper on PPO gives a nice description of the algorithm (the version with clipping the probability ratio is probably easier to understand): Proximal Policy Optimization Algorithms (Schulman et al., 2017)

OpenAI has a good description of general policy gradient algorithms and PPO as well, it's worth checking out.

You don't need to wait until the end of an episode to receive rewards. At least, not in the general sense. If you have access to intermediate rewards, then you can update the value network sooner. But PPO actually uses the advantage function which is also an n-step rollout or done via Generalised Advantage Estimation which still relies on bootstrapping from estimates at different timesteps.

Updated response to include more information from the discussion.

Monte Carlo vs. Temporal Difference (TD)

Let's start with the distinction between these two. When you have a sequence of rewards observed from the environment and a neural network predicting the value of each state, then you can create target values that your predictions should move closer to in a couple of ways. You can look at the full episode and use the actual observed rewards with discounting to create your target, this is called the Monte Carlo estimation. This target value is an estimation of the value function from your initial state.

$$ \widehat v_\pi(s_0) = y^{MC} = r_1 + \gamma r_2 + \gamma^2 r_3 + \cdot\cdot\cdot + \gamma^{T-1}r_T $$

Then, you update the NN parameters to get better at predicting the value of the state based on this estimate. Another way is to update your neural network sooner by only using a partial trajectory (of length $k$) and rely on your NN to estimate the rest of the trajectory.

$$ \widehat v(s_0) = y^{TD} = r_1 + \gamma r_2 + \gamma^2 r_3 + \cdot\cdot\cdot + \gamma^{k-1} r_k + \gamma^k \widehat v(s_k) $$

The last term uses the NN to estimate the remaining discounted rewards from the episode without observing it.

Proximal Policy Optimisation

You don't need to wait until the end of an episode to receive rewards. If you have access to intermediate rewards, then you can update the value network sooner. PPO uses the advantage function when calculating the objective (and the loss) which is also done similarly to the TD approach. Both the n-step and the Generalised Advantage Estimation relies on the NN to fill in some of the unobserved values.

Original paper on PPO gives a nice description of the algorithm (the version with clipping the probability ratio is probably easier to understand): Proximal Policy Optimization Algorithms (Schulman et al., 2017)

OpenAI has a good description of general policy gradient algorithms and PPO as well, it's worth checking out.

Source Link
user42664
user42664

You don't need to wait until the end of an episode to receive rewards. At least, not in the general sense. If you have access to intermediate rewards, then you can update the value network sooner. But PPO actually uses the advantage function which is also an n-step rollout or done via Generalised Advantage Estimation which still relies on bootstrapping from estimates at different timesteps.