As your reference mentioned below:
When the state-value function is used to assess actions in this way it is called a critic, and the overall policy-gradient method is termed an actor–critic method. Note that the bias in the gradient estimate is not due to bootstrapping as such; the actor would be biased even if the critic was learned by a Monte Carlo method.
Thus this inherent bias in actor-critic algo faces the same convergence issue as any TD boostrapping learning algo whose convergence is a theoretical result from stochastic approximation methods as discussed here.
converges with probability 1 to an optimal policy and action-value function, under the usual conditions on the step sizes (2.7), as long as all state–action pairs are visited an infinite number of times and the policy converges in the limit to the greedy policy... but not for the case of constant step-size parameter, $α_n(a)=α$. In the latter case, the second condition is not met, indicating that the estimates never completely converge but continue to vary in response to the most recently received rewards. As we mentioned above, this is actually desirable in a nonstationary environment, and problems that are effectively nonstationary are the most common in reinforcement learning.
But even this is the case it's is often desirable for the same reason that bootstrapping TD methods are often superior to Monte Carlo methods due to substantially reduced variance.