I am using RLlib (Ray 1.4.0)'s implementation of PPO for a multi-agent scenario with continuous actions, and I find that the loss includes the KL divergence penalty term, apart from the surrogate loss, value function loss, and entropy.
The KL coefficient is updated in the update_kl() function as follows:
if sampled_kl > 2.0 * self.kl_target: self.kl_coeff_val *= 1.5 # Decrease. elif sampled_kl < 0.5 * self.kl_target: self.kl_coeff_val *= 0.5 # No change. else: return self.kl_coeff_val
I don't understand the reasoning behind this. If the point of the KL "target" is to reach the target, then why do the conditions above imply that the KL coefficient is getting larger (multiplied by 1.5 when the sampled KL is already found to be larger than the target?) when it is supposed to be made smaller instead? I feel like I am missing something here, but I am not able to get my head around it.
I would appreciate any insights on this. Thank you.