4
$\begingroup$

In ''Proximal Policy Optimization Algorithms'' , Schulman et al. (2017), page 3

I don't understand why the clipped surrogate objective works. As written in the article : "With this scheme, we only ignore the change in probability ratio when it would make the objective improved, and we include it when it makes the objective worse". I feel confused : how can it works if it doesn't take account of objective improvements?

|improve this question
$\endgroup$
1
$\begingroup$

Ok, so I think I have a better understanding of this now.

Firstly, let's remind the main idea of the PPO : staying close to the previous policy. It's the same idea than in TRPO, but the L function is improved.

So, you wanna make "small but safe steps". With clipped surrogate objective, you don't give too much importance to promising actions. You learn that bad actions are bad, so you decrease their probability according to "how bad" they are. But for good actions, you only learn that they are "a little bit good", and their probability will be just slightly increased.

This mechanism allows you to perform small but relevant updates of your policy.

hope this will help someone :)

|improve this answer
$\endgroup$
  • $\begingroup$ Yes I think you understood it well. See Figure 1 on page 3. The red dots depict "where the policy" currently is. If we move down along the line (reduction in objective), we do not do any clipping at all. If we move up along the line (improvement in objective), we also do actually allow that for a little bit, but start clipping if we move too far from the existing policy. $\endgroup$ – Dennis Soemers Sep 7 '18 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.