I don't know what people mean by 'vanilla policy gradient', but what comes to mind is REINFORCE, which is the simplest policy gradient algorithm I can think of. Is this an accurate statement?

By REINFORCE I mean this surrogate objective

$$ \frac{1}{m} \sum_i \sum_t log(\pi(a_t|s_t)) R_i $$

Where I indices over the $m$ episodes and $t$ over time steps, and $R_i$ is the total reward of the episode. It's also common to replace $R_i$ with something else, like a baselined version $R_i - b$ or use the future return, potentially also with a baseline $G_{it} - b$. However, I think even with these modifications to the multiplicative term, people would still call this 'vanilla policy gradient'. Is that correct?

  • $\begingroup$ Use of the term may vary, depending on context, and what "extra" feature is being discussed. Could you give a reference to where someone uses "vanilla policy gradient"? With nothing else to go on, I think your understanding seems reasonable. $\endgroup$ Mar 27 '19 at 13:43
  • $\begingroup$ I'm thinking of TRPO and PPO right now, where the 'vanilla' part often implies that it's not sample efficient because it only performs one optimization step with the data. I can't point to other specific papers right now, but I think I've seen the term 'vanilla policy gradient' in other contexts too. Are there any other aspects of REINFORCE besides the single-optimization-update that makes it 'vanilla'? $\endgroup$
    – yewang
    Mar 27 '19 at 14:53

By vanilla policy gradient, I think what is meant is normally any arbitrary policy gradient for the purposes of formalization(or whatever is trying to be communicated).

For example, let $J(\theta)$ be any policy objective function. Then our policy gradient would be $\begin{equation} \nabla_\theta J(\theta) \end{equation}$ where the change in our policy is $\Delta \theta = \alpha \nabla_\theta J(\theta) $ with $\alpha$ being our step size.

This will, of course, vary as notational convention and term definition can change between practitioners.


OpenAI has a series of Spinning Up pages on their website to educate people about AI. One of those defines Vanilla Policy Gradiant.

Vanilla Policy Gradiant via OpenAI

At the bottom of the page are reference papers that further discuss gradiants.

Whether this is definitive for Vanilla Policy Gradients or not I do not know, but if many others refer to OpenAI for learning this subject their definition will spread.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.