How to implement PPO without using a Critic

I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my solution. From the original paper, it seems the clipped loss PPO optimizes includes an Advantage $$A_t$$ term:

$$L^{CLIP} (\theta) = \mathbb{E}_t \Big[min(r_t(\theta)A_t, \: clip(r_t(\theta), 1 - \epsilon, 1 + \epsilon)A_t \Big]$$

Generally, this advantage $$A_t$$ is calculated using an actor-critic schema, where we train a network (critic) to predict the value $$V(s)$$ of a given state, which is then used to calculate $$A_t$$. However, my RL task is episodic and the trajectories end (i.e., arrive at a terminal state) in just a few actions, so I would rather not use a critic network. Thus, my question is the following: How can I implement PPO without a critic network, i.e., without a network that predicts $$V(s)$$?

To achieve this, I thought about simply substituting the advantage term $$A_t$$ used in $$L^{CLIP}$$ for the discounted sum of rewards $$R_t$$ used in the REINFORCE loss. Should this work fine or is there a better alternative?

Your justification for not wanting to use a critic (that the episodes are short) does not make sense to me. I would expect that including the critic would result in a substantial variance reduction (and substantially faster training) due to the extra baseline and also the bootstrapping which is done in the generalized advantage estimation (GAE). I don't see how the episode length being short is relevant.

However, if you don't want to use an advantage, you don't have to. You can simply replace $$A_t$$ with the (monte carlo sampled) cumulative reward $$R_t$$. The problem is that the clipping objective only makes sense if you use an advantage. It won't make sense to do that clipping if you are using $$R_t$$ instead of $$A_t$$.

It seems dubious to me to call such an algorithm "PPO" as really it would just be REINFORCE. The only possible difference between what you describe and reinforce, is that PPO should generate a dataset of transitions and do multiple mini-batch updates per dataset. If you do decide you want to go that route, you should note that you still need the importance sampling term (the unfortunately named $$r_t(\theta)$$) for the update to be unbiased.

• The real reason why I don't want to use a critic is because I am using a specific NN architecture called Neural Logic Machine and it might be difficult to predict $V(s)$. As you mention, the reason why I want to use PPO instead of REINFORCE is to be able to increase the sample-efficiency of REINFORCE, by performing multiple gradient ascent iterations over the same data. However, if by using $R_t$ instead of the advantage then the clipping objective does not make sense, does this mean I could change the policy probabilities too much by performing multiple training iterations over the same data? Jun 12 at 15:36
• Yes it's possible that the training will be a bit more unstable without the clipping. But my biggest concern is that without the bootstrapping (which is included in gae) the algorithm may learn quite slowly
– Taw
Jun 12 at 16:12
• So, if I have understood you correctly, you think that even if the trajectories are short I should try to use actor-critic instead of vanilla REINFORCE and that will make a bigger impact in the sample-efficiency of my algorithm than using PPO (instead of REINFORCE) and performing multiple gradient ascents over the same data. Jun 12 at 16:43
• I am saying that the generalized advantage estimator (GAE) seems to be really good and I would highly recommend using it for the advantage calculation; yes this would require having a critic in addition to the policy. The question of sample efficiency is separate and it's hard to make any quantifiable claim there. But yes I believe it will definitely help.
– Taw
Jun 12 at 17:20