0
$\begingroup$

I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the following terms are equivalent -

This paper talks about Policy regularization using KL Divergence enter image description here

Whereas this paper uses the terms KL Stopping and KL Cutoff -

enter image description here

I think "Penalty" from the first paper is the same as "KL-cutoff". Also "Constraint" from the first paper is the same as "KL-Stopping". Could someone let me know if I am correct?

$\endgroup$

1 Answer 1

2
$\begingroup$

In my opinion both "Penalty" and "Constraint" have somewhat similar ideas to "KL-Cutoff", but "KL-Stopping" is entirely different.

The idea of policy regularization is to add some form of regularization to your loss function, much like adding L2-norm regularization in supervised learning. So for ppo you have:

ppo_loss = -torch.mean(torch.min(rho * adv, clip(rho, 1-eps, 1+eps) * adv))
loss = ppo_loss + alpha * reg

Now, the first paper discusses different types of regularization, e.g. entropy regularization or KL-divergence regularization. And also explains how to go about choosing the parameter alpha:

  • You could just treat it as a hyperparameter, just like you would treat the learning rate. (Penalty)
  • Or, you could optimize this parameter using a more sophisticated procedure. (Constraint)

You can add these regularization techniques to any policy gradient algorithm, e.g. vanilla policy gradient, actor-critic, or ppo.

KL-Cutoff and KL-Stopping from the second paper actually address a different problem. Using the ppo objective you are allowed to perform many policy updates before throwing away your data and performing new rollouts. But if you update too many times then your training might be unstable. So after every update you check the KL-divergence between the old policy (behavioral policy) and the current policy. If the KL-divergence reaches a given threshold, then you should do something:

  • KL-Stopping says to stop updating your policy, throw away the data, and rollout the new policy to collect new data.
  • KL-Cutoff suggests something very similar to the idea of policy regularization: continue updating your policy, but add the term $\pm \alpha(D_{KL} - D_{thr})^2$ to the loss function. As far as I understand the authors suggest a fixed parameter alpha, which is the same as "Penalty" from the fist paper. However, your regularizer is not the KL-divergence, but how much you are violating the threshold.

In my opinion these techniques are complementary. If you want to see this implemented in code I will add here a link to one of my repos where I implemented ppo:

  • here I use policy regularization with entropy,
  • here I use KL-stopping.
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .