How do I add Entropy to a PPO algorithm?

Question

I learned about adding entropy to RL algorithms through the notes provided in SpinningUp. They explained how entropy is added to the SAC algorithm. Here is my understanding - In entropy regularized RL, one adds an entropy bonus $H$ to the reward function. This changes the objective function of RL such that one needs to find a policy that'll maximize the expected sum of rewards and entropy.

In order to incorporate entropy into the actor (policy), one would have to maximize the q-function which contains the entropy. To incorporate entropy into the critic (value function), one would have to add entropy while computing the expected sum of future rewards. I hope this correct.

Now, I am trying to understand how to add entropy to the PPO algorithm which also has an actor and a critic. I think it'll be trivial to add entropy to add the critic. I just need to add entropy to the target return function while computing the loss between the predicted value and target. I have no idea how to incorporate entropy to the policy function though.

Answer 1

In the regular PPO formulation, which I don't believe is strictly following maximum-entropy RL, only the actor has the entropy term which is employed as a regularizer to favor exploration. (I guess adding $H$ to both actor and critic in PPO would result in a different agent.)

Basically, since the actor parameterizes a probability distribution like a Gaussian, whose entropy has an analytical form, you need to just compute that and add a term (weighted by a coefficient) to the loss function. If you don't know how to compute $H$, you can estimate it by taking the negative log-probability, averaging over the batch: $H[\pi]\approx -\frac1B\sum_{a,s\in B}-\log \pi(a\mid s)$. Note: the first minus is to ensure the optimizer to maximize the entropy.