The policy gradient states that
$$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a q_\pi(s, a) \nabla\pi(a | s; \theta)\;$$
where the derivatives are taken wrt the parameter $\theta$.
Now, if we say incorporate a baseline we get
$$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a \left( q_\pi(s, a) - b(s) \right)\nabla\pi(a | s; \theta)\;$$
and this does not effect the gradient at all. To see this, note that
$$\sum_a b(s) \nabla\pi(a|s; \theta) = b(s) \nabla \sum_a \pi(a|s; \theta) = b(s) \nabla 1 = 0\;;$$
where all I have done is expand the bracketed terms inside the sum over $a$ from the second equation, and shown that the new term is equal to 0 -- thus the gradient is unchanged.
If you really want to confirm this then you can fully write down the expansion of the second equation and use the trick I have shown you in my third equation to see that expanded second equation is equal to the first equation.
I imagine that the EGLP lemma that the authors referred to will use a similar trick of a derivative of a probability distribution equalling to 0 when summing(/integrating) over the support of the random variable, which is what I have used here to go from $\nabla \sum_a\pi(a|s; \theta) = \nabla 1$.