# Problems with gradient-biased actor critic methods

To my knowledge, there are at least 6 different variants of Actor Critic: $$\begin{array}{l l l l} \text{actor gradient} & \text{critic gradient} & \text{actor gradient biased} & \text{name} \\ \hline \sum_t \nabla_\theta \log \pi_\theta(a_t | s_t) \cdot (R_t - V_\theta) & \nabla_\theta \frac{1}{T} \sum_t (R_t - V_\theta(s_t))^2 & \text{no} & \text{Monte Carlo Advantage Actor Critic V (?)} \\ \sum_t \nabla_\theta \log \pi_\theta(a_t | s_t) \cdot (R_t - \sum_a \pi_\theta(a | s_t) Q_\theta(s_t, a)) & \nabla_\theta \frac{1}{T} \sum_t (R_t - Q_\theta(s_t, a_t))^2 & \text{no} & \text{Monte Carlo Advantage Actor Critic Q (?)} \\ \sum_t \nabla_\theta \log \pi_\theta(a_t | s_t) \cdot Q_\theta(s_t, a_t) & \text{same as above} & \text{yes} & \text{Actor Critic (?)} \\ \sum_t \nabla_\theta \log \pi_\theta(a_t | s_t) \cdot (Q_\theta(s_t, a_t) - \sum_a \pi_\theta(a | s_t) Q_\theta(s_t, a)) & \text{same as above} & \text{yes} & \text{Advantage Actor Critic (?)} \\ \sum_t \sum_a \nabla_\theta \pi_\theta(a | s_t) \cdot Q_\theta(s_t, a) & \text{same as above} & \text{yes} & \text{Mean Actor Critic} \\ \sum_t \sum_a \nabla_\theta \log \pi_\theta(a_t | s_t) \cdot (r_t + \gamma V(s_{t+1}) - V(s_t)) & \text{same as MCAACV} & \text{yes} & \text{TD Advantage Actor Critic (?)} \end{array}$$

The problem I'm having is that all methods which use a biased actor gradient are performing abysmally (even worse than their random initializations) on the simple CartPole task. See the graph below for an example:

I know that, ideally, we want $$Q_\theta$$ to reflect the true Q values under the current policy $$\pi_\theta$$, thus we probably want to "slow down" how quickly $$\pi_\theta$$ changes in order to give time for $$Q_\theta$$ to learn the true Q values of $$\pi_\theta$$. Methods like TRPO, PPO, etc. try to do this, but the Mean Actor Critic paper seems to get the biased-gradient methods (including their own) working without them (TRPO is separate from A2C and MAC in the Table 2). One thing I tried was scaling down the actor gradient in comparison to the critic gradient, but that didn't seem to help.

Does anyone know what the issue might be, and how to fix it?

• Hello. Please, put a more descriptive question in the title. "Problems with..." is not very descriptive.
– nbro
Oct 14, 2020 at 23:14
• @nbro What do you suggest? Oct 15, 2020 at 0:51
• Something like: "why do most of these gradient-based algorithms obtain a so low score in the cart pole environment", but note that I have only quickly read your question (and it was yesterday).
– nbro
Oct 15, 2020 at 7:44