4
$\begingroup$

In ''Proximal Policy Optimization Algorithms'' , Schulman et al. (2017), page 3

I don't understand why the clipped surrogate objective works. As written in the article : "With this scheme, we only ignore the change in probability ratio when it would make the objective improved, and we include it when it makes the objective worse". I feel confused : how can it works if it doesn't take account of objective improvements?

Improve this question
$\endgroup$
2
$\begingroup$

Ok, so I think I have a better understanding of this now.

Firstly, let's remind the main idea of the PPO : staying close to the previous policy. It's the same idea than in TRPO, but the L function is improved.

So, you wanna make "small but safe steps". With clipped surrogate objective, you don't give too much importance to promising actions. You learn that bad actions are bad, so you decrease their probability according to "how bad" they are. But for good actions, you only learn that they are "a little bit good", and their probability will be just slightly increased.

This mechanism allows you to perform small but relevant updates of your policy.

hope this will help someone :)

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes I think you understood it well. See Figure 1 on page 3. The red dots depict "where the policy" currently is. If we move down along the line (reduction in objective), we do not do any clipping at all. If we move up along the line (improvement in objective), we also do actually allow that for a little bit, but start clipping if we move too far from the existing policy. $\endgroup$
    – Dennis Soemers ♦
    Sep 7 '18 at 9:09
1
$\begingroup$

I think @16Aghnar explains the concept quite well. However, by clipping the surrogate objective alone doesn't ensure the trust region as stated in the paper:

Engstrom et al., 2020, Implementation Matters in Deep RL: A Case Study on PPO and TRPO.

The authors inspected OpenAI's implementation of PPO and find many code-level optimizations, I'll list the most important optimizations below:

  1. Clipped surrogate objective
  2. Value function clipping
  3. Reward scaling
  4. Orthogonal initialization and layer scaling
  5. Adam learning rate and annealing

They find that:

  • PPO-M (2.-5.) alone can maintain the trust region.

    (PPO-M: PPO without Clipped surrogate objective, but with code-level optimizations)

  • PPO-Clip (1. only) cannot maintain the trust region.

    (PPO-Clip: PPO without code-level optimizations, but with Clipped surrogate objective)

  • TRPO+ has better performance comparing to TRPO, and with similar performance comparing to PPO

    (TRPO+: TRPO with code-level optimizations used in PPO OpenAI implementation)

A intuitive thought on why Clipped surrogate objective alone does not work is: The first step we take is unclipped.

As a result, since we initialize $\pi_\theta$ as $\pi$ (and thus the ratios start all equal to one) the first step we take is identical to a maximization step over the unclipped surrogate reward. Therefore, the size of step we take is determined solely be the steepness of the surrogate landscape (i.e. Lipschitz constant of the ptimization problem we solve), and we can end up moving arbitrarily far from the trust region. -- Engstrom et al., 2020

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.