2
$\begingroup$

On the OpenAI's Spinning Up, they justify the fact that adding a baseline $b(s_t)$ in the policy gradient doesn't change its gradient by saying that this is

an immediate consequence of the EGLP Lemma

However, I did not manage to prove it with this lemma. Can somebody help me, please?

The proof is trivial when $b$ is a constant, but I struggle to derive it whenever $b$ is a function of the current state $s$ because you can't take it out of the integral.

$\endgroup$
2
$\begingroup$

The policy gradient states that $$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a q_\pi(s, a) \nabla\pi(a | s; \theta)\;$$ where the derivatives are taken wrt the parameter $\theta$.

Now, if we say incorporate a baseline we get $$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a \left( q_\pi(s, a) - b(s) \right)\nabla\pi(a | s; \theta)\;$$ and this does not effect the gradient at all. To see this, note that $$\sum_a b(s) \nabla\pi(a|s; \theta) = b(s) \nabla \sum_a \pi(a|s; \theta) = b(s) \nabla 1 = 0\;;$$ where all I have done is expand the bracketed terms inside the sum over $a$ from the second equation, and shown that the new term is equal to 0 -- thus the gradient is unchanged.

If you really want to confirm this then you can fully write down the expansion of the second equation and use the trick I have shown you in my third equation to see that expanded second equation is equal to the first equation.

I imagine that the EGLP lemma that the authors referred to will use a similar trick of a derivative of a probability distribution equalling to 0 when summing(/integrating) over the support of the random variable, which is what I have used here to go from $\nabla \sum_a\pi(a|s; \theta) = \nabla 1$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.