10
$\begingroup$

I don't know what people mean by 'vanilla policy gradient', but what comes to mind is REINFORCE, which is the simplest policy gradient algorithm I can think of. Is this an accurate statement?

By REINFORCE I mean this surrogate objective

$$ \frac{1}{m} \sum_i \sum_t log(\pi(a_t|s_t)) R_i,$$ where I index over the $m$ episodes and $t$ over time steps, and $R_i$ is the total reward of the episode. It's also common to replace $R_i$ with something else, like a baselined version $R_i - b$ or use the future return, potentially also with a baseline $G_{it} - b$.

However, I think even with these modifications to the multiplicative term, people would still call this 'vanilla policy gradient'. Is that correct?

$\endgroup$
2
  • $\begingroup$ Use of the term may vary, depending on context, and what "extra" feature is being discussed. Could you give a reference to where someone uses "vanilla policy gradient"? With nothing else to go on, I think your understanding seems reasonable. $\endgroup$ Mar 27, 2019 at 13:43
  • $\begingroup$ I'm thinking of TRPO and PPO right now, where the 'vanilla' part often implies that it's not sample efficient because it only performs one optimization step with the data. I can't point to other specific papers right now, but I think I've seen the term 'vanilla policy gradient' in other contexts too. Are there any other aspects of REINFORCE besides the single-optimization-update that makes it 'vanilla'? $\endgroup$
    – yewang
    Mar 27, 2019 at 14:53

3 Answers 3

4
$\begingroup$

You can check the Open AI Introduction to RL series, they explain pretty neatly there what is the Policy Optimization and how to derive it. I think, that usually when we are talking about REINFORCE algorithm, we are talking about the one described in Sutton's book on Reinforcement learning. It is described as the policy optimization algorithm maximizing the Value Function $v_{\pi(\theta)}(s) = E[G_t|S_t = s]$ of initial state of the agent. Here $G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1}$ is the $\gamma$ discounted return from given state, time $s, t$. Or, shortly put. $$ J(\theta) = v_{\pi(\theta)}(s_0) = E[G_t|S_t = s_0]\\ \nabla J(\theta) = E_\pi\left[G_t\frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}\right] $$ But in the RL series of Open AI, the algorithm that is described as Vanilla policy gradient (If it is the one you are talking about) is optimizing finite-horizon undiscounted return $E_{\tau \sim \pi} [R(\tau)] $, where $\tau$ are possible trajectories. e.g. $$ J(\theta) = E_{\tau \sim \pi} [R(\tau)] \\ \nabla J(\theta) = E_{\tau \sim\pi}\left[\sum_{t=0}^T R(\tau) \frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)}\right] $$

They do look very similar in objective functions, but they are different. The way the gradient ascent is performed differs strongly since in REINFORCE method the gradient ascent is performed once for each action taken for each episode and the direction of ascent is taken as

$$ G_t\frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)} $$ so the update becomes $$ \theta_{t+1} = \theta_{t} + \alpha G_t\frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)} $$ but in VPG algorithm the gradient ascents performed once over multiple episodes and direction of ascent taken as average

$$ \frac{1}{|\mathcal{T}|}\sum_{\tau\in\mathcal{T}} \sum_{t=0}^T R(\tau) \frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)} $$

and gradient ascent step is

$$ \theta_{t+1} = \theta_{t} + \alpha \frac{1}{|\mathcal{T}|}\sum_{\tau\in\mathcal{T}} \sum_{t=0}^T R(\tau) \frac{\nabla \pi (A_t |S_t, \theta)}{\pi (A_t |S_t, \theta)} $$ which looks a lot like what you have stated as REINFORCE algorithm.

I admit, that some form of mathematical equivalence can be derived between them, since expectation over policy and expectation over the trajectory sampled from the policy looks practically the same. But approaches differ at least in the way, the ascent is computed.

$\endgroup$
1
  • $\begingroup$ the answer is great. $\endgroup$
    – DehengYe
    Apr 13, 2022 at 3:42
1
$\begingroup$

OpenAI has a series of Spinning Up pages on their website to educate people about AI. One of those defines Vanilla Policy Gradiant.

Vanilla Policy Gradiant via OpenAI

At the bottom of the page are reference papers that further discuss gradiants.

Whether this is definitive for Vanilla Policy Gradients or not I do not know, but if many others refer to OpenAI for learning this subject their definition will spread.

$\endgroup$
1
$\begingroup$

By vanilla policy gradient, I think what is meant is normally any arbitrary policy gradient for the purposes of formalization (or whatever is trying to be communicated).

For example, let $J(\theta)$ be any policy objective function. Then our policy gradient would be $\begin{equation} \nabla_\theta J(\theta) \end{equation}$ where the change in our policy is $\Delta \theta = \alpha \nabla_\theta J(\theta) $ with $\alpha$ being our step size.

This will, of course, vary as notational convention and term definition can change between practitioners.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .