I don't understand why the clipped surrogate objective works. As written in the article : "With this scheme, we only ignore the change in probability ratio when it would make the objective improved, and we include it when it makes the objective worse". I feel confused : how can it works if it doesn't take account of objective improvements?
Ok, so I think I have a better understanding of this now.
Firstly, let's remind the main idea of the PPO : staying close to the previous policy. It's the same idea than in TRPO, but the L function is improved.
So, you wanna make "small but safe steps". With clipped surrogate objective, you don't give too much importance to promising actions. You learn that bad actions are bad, so you decrease their probability according to "how bad" they are. But for good actions, you only learn that they are "a little bit good", and their probability will be just slightly increased.
This mechanism allows you to perform small but relevant updates of your policy.
hope this will help someone :)