On the OpenAI's Spinning Up, they justify the fact that adding a baseline $$b(s_t)$$ in the policy gradient doesn't change its gradient by saying that this is

an immediate consequence of the EGLP Lemma

However, I did not manage to prove it with this lemma. Can somebody help me, please?

The proof is trivial when $$b$$ is a constant, but I struggle to derive it whenever $$b$$ is a function of the current state $$s$$ because you can't take it out of the integral.

The policy gradient states that $$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a q_\pi(s, a) \nabla\pi(a | s; \theta)\;$$ where the derivatives are taken wrt the parameter $$\theta$$.
Now, if we say incorporate a baseline we get $$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a \left( q_\pi(s, a) - b(s) \right)\nabla\pi(a | s; \theta)\;$$ and this does not effect the gradient at all. To see this, note that $$\sum_a b(s) \nabla\pi(a|s; \theta) = b(s) \nabla \sum_a \pi(a|s; \theta) = b(s) \nabla 1 = 0\;;$$ where all I have done is expand the bracketed terms inside the sum over $$a$$ from the second equation, and shown that the new term is equal to 0 -- thus the gradient is unchanged.
I imagine that the EGLP lemma that the authors referred to will use a similar trick of a derivative of a probability distribution equalling to 0 when summing(/integrating) over the support of the random variable, which is what I have used here to go from $$\nabla \sum_a\pi(a|s; \theta) = \nabla 1$$.