On the OpenAI's Spinning Up, they justify the fact that adding a baseline $b(s_t)$ in the policy gradient doesn't change its gradient by saying that this is
an immediate consequence of the EGLP Lemma
However, I did not manage to prove it with this lemma. Can somebody help me, please?
The proof is trivial when $b$ is a constant, but I struggle to derive it whenever $b$ is a function of the current state $s$ because you can't take it out of the integral.
The policy gradient states that $$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a q_\pi(s, a) \nabla\pi(a | s; \theta)\;$$ where the derivatives are taken wrt the parameter $\theta$.
Now, if we say incorporate a baseline we get $$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a \left( q_\pi(s, a) - b(s) \right)\nabla\pi(a | s; \theta)\;$$ and this does not effect the gradient at all. To see this, note that $$\sum_a b(s) \nabla\pi(a|s; \theta) = b(s) \nabla \sum_a \pi(a|s; \theta) = b(s) \nabla 1 = 0\;;$$ where all I have done is expand the bracketed terms inside the sum over $a$ from the second equation, and shown that the new term is equal to 0 -- thus the gradient is unchanged.
If you really want to confirm this then you can fully write down the expansion of the second equation and use the trick I have shown you in my third equation to see that expanded second equation is equal to the first equation.
I imagine that the EGLP lemma that the authors referred to will use a similar trick of a derivative of a probability distribution equalling to 0 when summing(/integrating) over the support of the random variable, which is what I have used here to go from $\nabla \sum_a\pi(a|s; \theta) = \nabla 1$.
