How to implement PPO without using a Critic

Question

I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my solution. From the original paper, it seems the clipped loss PPO optimizes includes an Advantage $A_t$ term:

$$ L^{CLIP} (\theta) = \mathbb{E}_t \Big[min(r_t(\theta)A_t, \: clip(r_t(\theta), 1 - \epsilon, 1 + \epsilon)A_t \Big] $$

Generally, this advantage $A_t$ is calculated using an actor-critic schema, where we train a network (critic) to predict the value $V(s)$ of a given state, which is then used to calculate $A_t$. However, my RL task is episodic and the trajectories end (i.e., arrive at a terminal state) in just a few actions, so I would rather not use a critic network. Thus, my question is the following: How can I implement PPO without a critic network, i.e., without a network that predicts $V(s)$?

To achieve this, I thought about simply substituting the advantage term $A_t$ used in $L^{CLIP}$ for the discounted sum of rewards $R_t$ used in the REINFORCE loss. Should this work fine or is there a better alternative?

Answer 1

Your justification for not wanting to use a critic (that the episodes are short) does not make sense to me. I would expect that including the critic would result in a substantial variance reduction (and substantially faster training) due to the extra baseline and also the bootstrapping which is done in the generalized advantage estimation (GAE). I don't see how the episode length being short is relevant.

However, if you don't want to use an advantage, you don't have to. You can simply replace $A_t$ with the (monte carlo sampled) cumulative reward $R_t$. The problem is that the clipping objective only makes sense if you use an advantage. It won't make sense to do that clipping if you are using $R_t$ instead of $A_t$.

It seems dubious to me to call such an algorithm "PPO" as really it would just be REINFORCE. The only possible difference between what you describe and reinforce, is that PPO should generate a dataset of transitions and do multiple mini-batch updates per dataset. If you do decide you want to go that route, you should note that you still need the importance sampling term (the unfortunately named $r_t(\theta)$) for the update to be unbiased.