# Is REINFORCE the same as 'vanilla policy gradient'?

I don't know what people mean by 'vanilla policy gradient', but what comes to mind is REINFORCE, which is the simplest policy gradient algorithm I can think of. Is this an accurate statement?

By REINFORCE I mean this surrogate objective

$$\frac{1}{m} \sum_i \sum_t log(\pi(a_t|s_t)) R_i$$

Where I indices over the $$m$$ episodes and $$t$$ over time steps, and $$R_i$$ is the total reward of the episode. It's also common to replace $$R_i$$ with something else, like a baselined version $$R_i - b$$ or use the future return, potentially also with a baseline $$G_{it} - b$$. However, I think even with these modifications to the multiplicative term, people would still call this 'vanilla policy gradient'. Is that correct?

• Use of the term may vary, depending on context, and what "extra" feature is being discussed. Could you give a reference to where someone uses "vanilla policy gradient"? With nothing else to go on, I think your understanding seems reasonable. Mar 27 '19 at 13:43
• I'm thinking of TRPO and PPO right now, where the 'vanilla' part often implies that it's not sample efficient because it only performs one optimization step with the data. I can't point to other specific papers right now, but I think I've seen the term 'vanilla policy gradient' in other contexts too. Are there any other aspects of REINFORCE besides the single-optimization-update that makes it 'vanilla'? Mar 27 '19 at 14:53

By vanilla policy gradient, I think what is meant is normally any arbitrary policy gradient for the purposes of formalization(or whatever is trying to be communicated).

For example, let $$J(\theta)$$ be any policy objective function. Then our policy gradient would be $$$$\nabla_\theta J(\theta)$$$$ where the change in our policy is $$\Delta \theta = \alpha \nabla_\theta J(\theta)$$ with $$\alpha$$ being our step size.

This will, of course, vary as notational convention and term definition can change between practitioners.

OpenAI has a series of Spinning Up pages on their website to educate people about AI. One of those defines Vanilla Policy Gradiant.